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ABSTRACT 
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INTRODUCTION 



CONFERENCE PROCEEDINGS HAVE BEEN AN ANNUAL FEATURE 
SINCE 1958. PRIOR TO THAT DATE, AN ANNUAL REPORT 
WAS GIVEN ABOUT THE CONFERENCE AFTER ITS CONCLUSION. 

EACH PARTICIPANT THIS YEAR SUBMITTED HIS REMARKS ON 
SPECIALLY DESIGNED PAPER. EACH PAGE WAS PHOTOGRAPHED 
IN ORDER TO PREPARE A PLATE SO THAT THE PROCEEDINGS 
COULD BE PRINTED AS SUBMITTED BY THE AUTHORS, WITHOUT 
EDITING BY THE CTA STAFF. 

CTA ASSISTANT RESEARCH EXECUTIVE, DONALD P. GLASER, IS 
CONFERENCE PROGRAM MANAGER AND JOSEPHINE BROCK IS 
CONFERENCE PROGRAM SECRETARY. 
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FIRST GENERAL SESSION 



SPEAK.ER: LEE J. CRONBACH 

VIDA JACKS PROFESSOR OF EDUCATION 
STANFORD UNIVERSITY 

TOPIC: "WHAT CAN GO WRONG IN EVALUATION?*' 



WHAT CAN GO WRONG IN EVALUATION? 



Lee J. Cronbach 



(Lee J. Cronbach is Vida Jacks Professor of Education at Stanford 
University. He received his B.A. degree from Fresno State 
College, his M.A. from the University of California at 
Berkeley and his Ph.D. from the University of Chicago.) 

The term evaluation is now much used, and evaluations are in great 
demand. The term is ambiguous, because there are many kinds of evalua- 
tion. Too often, there are misunderstandings because legislators, 
educators, and the public expect one kind .of inquiry to serve all pur- 
poses. Professionals misapply principles of evaluation because they 
do not realize that the validity of the principle is limited to certain 
kinds of investigation. 

Evaluation is the collection and analysis of data that will help 
a ciacision maker. Too often it is considered to be an entirely scien- 
tific inquiry, but it is a social process, and a means of educating 
the decision makers. Evaluation is not passive, neutral, or value-* 
free. The evaluatcr has responsibility to improve the program he 
evaluates, not merely to turn in a report card. 

The data from which evaluations are made are often too limited. 
This is especially the case when evaluations are dominated by objective 
tests and by narrowly conceived ''behavioral objectives". The essence of 
quality ±xi an educational program is its ability to improve performance 
on transfer tests, i.e., on matters not studied. Evaluations that range 
beyond the prescribed curricular content can threaten teachers and 
students, but they are indispensible . Attention needs to be given to 
those outcomes that arise out of the whole educational process, not being 
the domain of any one course or subject field. 

The analysis of evaluative data is often too simple. There are 
limits to the depth of an analysis a target audience can profit from, 
but one should run the risk of analyzing more deeply rather than less 
deeply. 

Good evaluation is costly. The person planning an evaluation 
must make hard choices to decide what information will have the great- 
est effect on program quality. He must prepare a report that will be 
meaningful to the decision maker » and educationally valid. No standard 
rules can tell him how to accomplish this in any one setting. 
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SECOJro GENERAL SESSION 



SPEAKER: HARRY F. SILBERMAN 
TASK FORCE LEADER 
NATIONAL INSTITUTE OF EDUCATION 

TOPIC: "PROGRAM PLANNING FOR NIE" 
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Dr. Harry F. Silbennan 



PROGRAM PLANNING FOR NIE 

The NIE arid the Office of Education are now co-equal branches of 
the new Education Division in HEW created by the Education Amendments 
of 1972. 

The NIE legislation specified that the Institute shall seek to 
improve education, including career education, in the United States. 
To achieve its purpose, NIE must be able to identify the goals of American 
education, but in a pluralistic society it is difficult to specify a set 
of goals with which everyone will agree. Nevertheless, some attempts have 
been made in our studios to describe a set of goals on which there is some 
agreement and we considered the problems most frequently cited as obsta- 
cles to achieving those goals. The problems most frequently cited were: 

(1) Access; (2) Participation; (3) Productivity; (4) Substance. 

The program suggestions that are most often proposed to overcome 
these obstacles require that certain factors be changed; (1) Laws; 

(2) Incentives; (3) Information; (4) Agents. 

NIE R&D programs call for substantial experimentation with these 
four variables. Program plans reflect both new initiatives and NIE's 
responsibility to assume management of certain on-going research and 
development activities previously supported by the Office of Education. 

Some of the on-going programs that were transferred from the Office 
of Education include: The National Center for Educational Communications, 
the Experimental Schools Programs, the Career Education Program, the 
Regional Laboratory and R&D Centers Program, the Researcher Training 
Program, and the Unsolicited Research Program. These programs comprise 
approximately 807o of the National Institute of Education's budget. 

The program areas in which planning for new initiatives is pro- 
ceeding are as follows: 1. Home-based Education; 2. School-Work 
Transition; 3. Community Participation; 4. Post Secondary Alternatives; 
5. School Finance; 6. New Measures for Education; 7. Self Directed 
Education; 8. Productivity. 

An important share of NIE's attention should be devoted to broad 
policy issties related to education. Much of this work may be done 
intramurally as part of a "think tank" operation in which senior staff 
on temporary leave from their home institutions spend their time at NIE 
in an attenqjt to synthesize what is known about crucial issues and make 
recommendations to governmental officials on matters pertaining to 
education. 

There are several important outcomes from such syBtematization and 
analysis of R&D findings. Without an explicit effort to organize what 
we have learned from past R&D effort, we shall continue to travel the 
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same cul-de-sacs. An institutionalized memory of past failures will 
conserve NIE resources so that new program plans will be more likely 
to have a cumulative effect. Another possible outcome of a systematic 
NIE attempt to synthesize R&D is more realistic advice for legislators 
and executives at Federal, State, and local levels, who must make 
decisions on educational policy. Where solutions to problems are not 
available, where knowledge gaps prevent analysis and existing RfcD 
activities are not already underway, alternative program plans will be 
prepared for new NIE research initiatives. Thus, NIE might support 
directed studies on such topics as school finance, organization reform, 
or accreditation and certification practices. Longitudinal studies of 
entire communities might also be undertaken to assess the relative 
influence of major determinants of educational achievement, such as 
family, mass media, peer culture, employer practices, schooling, etc. 



8 



THIRD GENERAL SESSION 



SPEAKER: ASSEMBLYMAN JOHN STULL 
80th ASSEMBLY DISTRICT 
STATE OF CALIFORNIA 

TOPIC: "TEACHER COMPETENCY ACT, AB 293" 
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TEACHER COMPETENCY ACT, AB 293 



ASSEMBLYMAN JOHN STULL 
80th ASSEMBLY DISTRICT 
STATE OF CALIFORNIA 
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LADIES AND GENTLEMEN: 

I APPRECIATE HAVING THIS OPPORTUNITY TO SHARE SOME THOUGHTS 
WITH YOU ON MY BILL ON CALIFORNIA TEACHER TENURE— A MATTER WHICH 
SHOULD BE OF VITAL CONCERN TO ALL OF US/ AS SCHOOL ADMINISTRATORS/ 
AS EDUCATORS/ AS PARENTS/ AS TAXPAYERS/ AND AS CITIZENS OF THIS 
STATE, 

Our whole situation reminds me of the old story about the 
doctor/ the engineer/ and the politician who were arguing about 

which of THEIR PROFESSIONS WAS THE OLDEST, ThE DOCTOR SAID THAT/ 

of course/ medicine was the oldest/ that mankind had always had 
physicians/ and that they are even mentioned in the bible, 

"That's nothing/" said the Engineer/ "the Bible tells us how 

THE world was CREATED OUT OF CHAOS/ AND HOW COULD THERE BE ANY 

order brought out of chaos without an engineer to help?" 
Whereupon the politician broke in, "Wait a minute/" he 

SAID/ " who do you think CREATED THE CHAOS?" 

Well^ I'll admit to being a politician myself/ but I don't 
like the current chaos any better than the rest of you, That's 
one of the reasons that passage of AB 293 provides much satisfaction- 
it's A bright note, 

I HAVE SERVED IN THE LEGISLATURE NOW FOR OVER SIX YEARS/ 
AND I HAVE BEEN REQUESTED TIME AFTER TIME TO PROVIDE MORE FUNDS 
FOR SCHOOLS, I HAVE BEEN ASKED TO ALTER THE CREDENTIALING LAWS/ 
LOWER CLASS SIZE/ INCREASE THE NUMBER OF NURSES/ PSYCHOLOGISTS/ 
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AND COUNSELORS/ RAISE TEACHER SALARIES AND EVEN TO ELIMINATE 
THE PITHING OF LIVE ANIMALS, NoW FOR EACH OF THESE REQUESTS 
THERE HAS BEEN A REASONED ARGUMENT DESIGNED TO DEMONSTRATE THE 
ESSENTIALITY OF THE CAUSE. BUT I WANT YOU TO KNOW THAT IN NO 
CASE WAS EVIDENCE PRESENTED WHICH WOULD CLEARLY DEMONSTRATE THAT 
STUDENTS WOULD LEARN MORE WITH ANY OF THESE CHANGES, 

I HAVE FROM TIME TO TIME SUPPORTED THE REQUESTS FOR INCREASES 
IN EACH OF THE ABOVE SERVICES. AnD I HAVE FROM TIME TO TIME 
OPPOSED SUCH REQUESTS. BUT WITH EACH PASSING YEAR I BEGAN MORE AND 
MORE TO QUESTION THE BASIS FOR EACH REQUEST. 

What I always wanted to know was what will be the benefit 

TO CHILDREN FROM SUCH ALLOCATIONS OF DOLLARS. WhAT I CARE 

about is not so much how services are delivered but rather 
how well are trley received. i want the governmental process to 
concentrate on the output rather than on the input, 

What AB 293 attempts to do is to focus the attention of the 
school community on the progress of children. i do not believe 
that it ohould be the function of the legislature to tell 
professionals how to provide services/ but professional flexi- 
bility—often called academic freedom— requi res as its counter- 
weight professional accountability based upon the product of 
the professional service, so ab 293 attempts to concentrate 
attention upon the product of the educational system/ through 
evaluation, 

To ME AB 293 WAS a first attempt to create an evaluation 

SYSTEM COUPLED WITH A REVISED DUE-PROCESS SYSTEM WHICH WILL 

AFFIRM OR DENY THE EFFICACY OF TH^E EVALUATION SYSTEM. ThAT IS 
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REALLY A COMPLEX SENTENCE/ BUT IN TRUTH THIS IS THE ESSENCE OF 
AB 293. 

To SPEAK FAVORABLY OF TEACHER EVALUATION IS TO VENTURE INTO 
HAZARDOUS TERRITORY, EVALUATION OF SCHOOL PERSONNEL HAS NEVER 
BEEN POPULAR, ThE EDUCATION PROFESSION HAS RESISTED/ AVOIDED/ 

rationalized/ or watered down nearly all efforts to confront the 
issue of evaluation on open analytical grounds, 

The wisdom with which any evaluation system is implemented 

WILL HAVE much TO DO WITH NOT ONLY THE PROMISE OF HIGHER- QUALITY 
EDUCATION FOR OUR CHILDREN/ BUT/ INDEED/ THE VIEWS OF SOCIETY 
TOWARD EDUCATION IN GENERAL, ThE OVERAl L PURPOSE OF EVALUATION 
IN AB 293 IS INSTRUCTIONAL IMPROVEMENT, It IS TO HELP EACH 
TEACHER AND EACH ADMINISTRATOR SUCCEED IN HIS OR HER MISSION TO 
OPEN OUR WORLD TO CHILDREN, 

ADMITTEDLY/ THE IMPLEMENTATION OF THE EVALUATION CALLED FOR 
IN AB 293 WILL TAKE TIME/ RESOURCES/ AND TALENT— VERY LIKELY 
MORE THAN IS AVAILABLE TODAY— BUT IT IS IMPERATIVE THAT WE MOVE 
NOW TO BEGIN, 

NoW/ I AM AWARE THAT THE EVALUATION PROVISIONS OF AB 293 
HAVE CAUSED A GREAT DEAL OF CONSTERNATION WITHIN THE EDUCATIONAL 
PROFESSION, I HAVE BEEN TOLD THAT AB 293 IS RIGID/ BUREAUCRATIC/ 
AND EXTREMELY UNREALISTIC, I HAVE BEEN TOLD THAT AB 293 REQUIRES 
THE USE OF STANDARDIZED TESTS TO MEASURE PUPILS ' PROGRESS, I 
HAVE BEEN CASTIGATED BY THOSE WHO BELIEVE THAT AB 293 IS SUBTERFUGE 
FOR THE IMPLEMENTATION OF PPBS/ AND I HAVE BEEN ACCUSED OF BEING 
PUNITIVE TOWARDS TEACHERS/ IGNORANT OF EDUCATIONAL PROCESSES/ 
AND ANTI PUBLIC SCHOOLS, 
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WELL/ NOBODY IS PERFECT, 

What I believe AB 293 was designed to do is to set about a 
process which will require a closer examination of the progress 
of students and the functions of those paid to promote such 
progress in the public schools. 

Initial legislative studies on the subject of tenure 
began in 1968 with the senate education committee reviewing 
California's tenure structure during interim studies. In 1969/ 
THE Assembly Subcommittee on Educational Environment/ which I 

CHAIRED/ reviewed TENURE IN ELEMENTARY/ SECONDARY/ AND HIGHER 

education. In 1970 and 1971 legislation resulting in AB 293 was 

REVIEWED. 

In both THE Assembly and Senate studies of existing tenure 

LAWS/ THE EVALUATION SYSTEM WAS OF KEY CONCERN. EVEN IN 
DISCUSSING DISMISSAL PROCEDURES/ ONE CONCEPTUAL QUESTION WHICH 
ALWAYS AROSE WAS/ "HoW DO YOU ASSESS COMPETENCY?" IN AB 293 I 

feel the legislature has provided you/ the educational community/ 
with the guidelines necessary for a reasonable assessment of 
competency. i would/ therefore/ now like to review with you 
the evaluation section of the new law. 

Among other thingS/ the law requires that each school board 
adopt a uniform set of evaluation and assessment guidelines for 
use in evaluating the professional competency of its certificated 
personnel/ and further requires that these guidelines be put 

IN WRITING. In other WORDS/ SCHOOLS BOARDS ARE/ UNDER THE NEW 
LAW/ REQUIRED TO PUT DOWN ON PAPER AND MAKE AVAILABLE TO THEIR 

CERTIFICATED PERSONNEL GUIDELINES OF EXPECTED JOB PERFORMANCE/ 
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GUIDELINES DESIGNED TO ASSIST THE CERTIFICATED PERSON IN KNOWING 
WHAT IS EXPECTED OF HIM AND HOW THE SCHOOL BOARD WILL GO ABOUT 
REVIEWING HIS PERFORMANCE, 

In the development of these evaluation and assessment GUIDE- 
LINES AND PROCEDURES/ THE NEW LAW REQUIRES THAT SCHOOL BOARDS 
AVAIL THEMSELVES OF THE ADVICE OF CERTIFICATED INSTRUCTIONAL 
PERSONNEL. It WAS MY INTENTION FROM THE OUTSET THAT CERTIFICATED 
PERSONNEL BE REVIEWED BY THEIR PEERS, To ENSURE THAT THE 
CRITERIA ESTABLISHED ARE APPROPRIATE TO THE FIELD IN WHICH THE 
EMPLOYEE IS CERTIFICATED/ I FELT THAT INPUT FROM THE CERTIFICATED 
PERSONNEL THEMSELVES IN THE FORMULATIVE STAGES WAS ESSENTIAL, 
In WRITING THIS PROVISION INTO THE LAW/ I ENVISIONED THIS INVOLVE- 
MENT BEING ACCOMPLISHED EITHER THROUGH A TEACHERS' COMMITTEE/ 
APPOINTED BY THE SUPERINTENDENT/ OR WITH THE ASSISTANCE OF THE 
CERTIFICATED EMPLOYEE COUNCIL. No MATTER WHICH METHOD IS IN 
FACT UTILIZED/ CERTIFICATED PERSONNEL INPUT AT THE FORMULATIVE 
LEVEL IS ESSENTIAL TO ENSURE A QUALITY PRODUCT/ AND I AM HOPEFUL 
THAT YOU WILL KEEP THIS PORTION OF THE NEW LAW IN MIND AS YOU 
PROCEED WITH ITS IMPLEMENTATION, 

In FORMULATING EVALUATION AND ASSESSMENT GUIDELINES/ THE 
NEW LAW CLEARLY ESTABLISHES FOUR AREAS THAT MUST BE CONSIDERED: 

1, The ESTABLISHMENT OF STANDARDS OF EXPECTED 

student progress in each area of study and techniques 
for the assessment of that progress; 

2. Assessment of certificated personnel com- 
petence AS IT relates to THE ESTABLISHED STANDARDS; 
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3, Assessment of other duties normally required 
TO be performed by certificated employees as an adjunct 

TO THEIR regular DUTIES AND ASSIGNMENTS; AND/ 

^. The establishment of procedures and techniques 

FOR ascertaining THAT THE CERTIFICATED EMPLOYEE IS 
MAINTAINING PROPER CONTROL AND IS PRESERVING A 
SUITABLE LEARNING ENVIRONMENT. 

Now IN LEGAL TERMS/ THESE FOUR POINTS ARE/ I GUESS/ EASILY 
UNDERSTOOD. BUT IN LAYMAN TERMS/ SUCH AS YOU AND I CAN UNDER- 
STAND/ WHAT IS THIS PORTION OF THE NEW LAW SAYING? 

BASICALLY/ THE FIRST TWO POINTS REQUIRE YOU TO ESTABLISH AN 
EXPECTED LEVEL OF STUDENT PROGRESS OVER A GIVEN PERIOD OF TIME 
AND THEN ADOPT A SYSTEM OF EVALUATION TO SEE IF PROGRESS HAS 
BEEN ACHIEVED. YoU ARE THEN ASKED TO EVALUATE THE COMPETENCY OF THE 
CERTIFICATED EMPLOYEE ON THE BASIS OF HOW WELL HIS STUDENTS 
PROGRESSED. In OTHER WORDS/ WHAT DO YOU EXPECT THE STUDENT 
TO LEARN/ AND HAVE EMPLOYEES DONE THEIR BEST TO BRING THE 
STUDENT ALONG SATISFACTORILY TO MEET THAT EXPECTATION? 

The REMAINING TWO POINTS INVOLVE JUDGING A CERTIFICATED 
employee's performance in PROFESSIONAL WORK RELATED TO HIS 

primary normal assignments/ and his ability to maintain 
discipline and good order among his students. 

It has come to my attention that many administrators have 
interpreted this fourth point— pertaining to the preservation 
of proper control and a suitable learning environment— in such 

A WAY THAT IT APPLIES SOLELY TO INSTRUCTORS. ThIS IS CLEARLY 

NOT THE INTENT OF THIS POINT/ NOR IS IT THE INTENT OF THE LAW. 
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This law applies to all certificated employees/ from superintendent 
ON DOWN, Every certificated employee of a school district is 
responsible in his own area to ensure that every student is 
afforded the best possible education by the best qualified 
certificated personnel, This was my intent, and I believe 
IT is clearly the intent of the law. 

Although these four points constitute in the minds of 
many educators the heart of the law/ of equal importance to 
me is the requirement that any guidelines which are established 
must include adequate provisions for follow-up counseling, more 
specifically/ this new law requires you to take four specific 
steps before any dismissal action can be taken against a 
certificated employee considered deficient in his evaluation 

RATINGS: 

1, The CERTIFICATED EMPLOYEE MUST BE NOTIFIED 

in writing of the unsatisfactory performance; 

2, The written notice of unsatisfactory per- 
formance MUST BE clearly AND COMPREHENSIVELY STATED; 

3, The evaluator must meet and discuss with the 
employee and make specific recommendations on improve- 
ment of his performance; and/ 

^, Positive steps must be taken to assist the 
employee in upgrading his performance. 
All too often professional personnel are not given the 

OPPORTUNITY to HAVE THEIR WORK REGULARLY EVALUATED AND/ IN TURN/ 
GIVEN THE OPPORTUNITY TO IMPROVE IN AREAS IN WHICH THEY MAY BE 
DEFICIENT. This section of the NEW LAW NOT ONLY PROVIDES THIS 
OPPORTUNITY— IT MANDATES IT, 
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In this CONTEXT/ THEN/ I DO NOT FIND THE EVALUATION PROVISIONS 

of ab 293 anything more than a codification of what sound 
educational practice has been for many years. 

i am fully aware that the implementation of these policies 
has required and will continue to require a great amount of 
diligent work, 

It may be that carrying out these evaluation procedures 
will require changing the role of many existing educational 
professionals, 

It may be that the establishment of educational standards 

WILL cause some SERIOUS RETHINKING OF EDUCATIONAL OBJECTIVES. 

It may BE that carrying out the EVALUATION PROCESS WILL 
REQUIRE REARRANGEMENT OF RESOURCE ALLOCATIONS. 

If such is the casE/ I will be pleased; for these considera- 
tions ALONE WILL STRONGLY INDICATE THAT WE ARE REFOCUSING OUR 
ATTENTION ON THE OUTCOMES OF OUR ACTIONS. 

I HAVE BEEN TOLD SO OFTEN BY EDUCATORS THAT IT IS IMPOSSIBLE 
TO EFFECTIVELY EVALUATE PROFESSIONAL PERSONNEL. YeT/ EVERY 
TIME I GO TO A SCHOOL I CAN RECEIVE/ PRIVATELY/ A LIST OF THE 
FIVE BEST AND FIVE WORST PROFESSIONALS IN THE SCHOOL. AnD/ I 
MIGHT ADD/ THE LIST IS PRETTY MUCH THE SAME WHETHER IT COMES 
FROM TEACHERS OR ADMINISTRATORS. ThE EVALUATION SECTION OF AB 293 
ASKS THAT THOSE JUDGMENTS BE ACTED UPON EITHER TO MAKE THE 
WORST BETTER OR TO HELP THEM FIND OTHER EMPLOYMENT. 

The dismissal or due-process procedures of AB 293 are an 

ATTEMPT TO ESTABLISH A MORE PROFESSIONAL REVIEW OF CASES WHERE 
ATTEMPTS AT UPGRADING HAVE ALLEGEDLY NOT BEEN SUCCESSFUL. 
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Previous law required thai if a local district wished to 

DISMISS A certificated TENURED EMPLOYEE IT WOULD BE REQUIRED 
TO FILE A CASE IN SUPERIOR COURT. TeNURED CERTIFICATED EMPLOYEES 
WERE THE ONLY CLASS OF PROFESSIONALS FOR WHICH SUCH ACTION 
WAS REQUIRED. It WAS OUR FEELING THAT SUCH A PROCEDURE DID 
NOT CONTAIN ENOUGH PROFESSIONAL INVOLVEMENT, 

The DUE-PROCESS procedures in AB 293 provide that if a 

CASE involves QUESTIONS OF FITNESS FOR SERVICES (INCOMPETENCY/ 

dishonesty/ wilful refusal to obey school laws and regulations/ 
etc.)/ it will be decided by a three-member panel. 

we refer to this body as the commission on professional 
Competency. The Commission on Professional Competency shall 
sit when charges involve the qualifications of the certificated 
employee to continue in his designated capacity/ based upon 
the evaluation and assessment guidelines established by the 

LOCAL DISTRICT. ThE COMMISSION WILL CONSIST OF THREE MEMBERS — 
THE HEARING OFFICER AND TWO EXPERIENCED PERSONNEL WHO SHALL HAVE 
AT LEAST FIVE YEAR'S EXPERIENCE IN THE SPECIFIC EDUCATIONAL 

function of the accused/ as set forth in section 13055 of the 
Education Code. One of the two shall be chosen by the employing 

AUTHORITY AND THE OTHER BY THE EMPLOYEE IN QUESTION, ThE HEARING 

OFFICER FROM THE OFFICE OF ADMINISTRATIVE PROCEDURE SHALL 

SERVE AS CHAIRMAN AND A VOTING MEMBER OF THE THREE-MAN PANEL, 

In THOSE CASES REQUIRING A COMMISSION ON PROFESSIONAL 

COMPETENCE/ THE PANEL SHALL/ BY A MAJORITY VOTE/ PREPARE A 

WRITTEN DISPOSITION CONTAINING THE FINDINGS OF. FACT/ DETERMINATIONS 

OF ISSUES/ AND A DECISION EITHER TO RETAIN OR TO DISMISS THE 
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EMPLOYEE, Its decision shall be deemed to be the final decision 

OF THE SCHOOL DISTRICT'S GOVERNING BOARD, In CASES HEARD ONLY 

before the hearing officer/ his decision shall also be binding 
on the school board, 

The hearing officer alone will consider all charges dealing 
with acts of criminal syndicalism/ physical or mental conditions 
which affect ability to perform/ conviction of a felony or crime of 
moral turpitude/ violations of specific codes/ or knowing 

MEMBERSHIP IN THE COMMUNIST PaRTY , 

Either party maY/ of course/ appeal the decision to the 
Superior Court, In including this appeal procedure/ howeveR/ 
it was not then/ nor is it now/ my intention to allow the 

courts TO HAVE TRIAL HE NOVO IN HEARING THE CASE, ALTHOUGH THE 
LAW DOES READ THAT THE COURTS MAY EXERCISE THEIR "INDEPENDENT 
JUDGMENT ON THE EVIDENCE/" IT ALSO STATES THAT THE COURTS 
SHALL REVIEW ANY APPEALS IN THE SAME MANNER AS A DECISION MADE 
BY A HEARING OFFICER UNDER CHAPTER 5 OF THE GOVERNMENT CODE, 

So, AB 293 REALLY IS JUST A MANDATE OF AN EVALUATION 
SYSTEM COMBINED WITH CIRCUMSPECT CHANGES IN A DUE-PROCESS SYSTEM 
WHICH WOULD BE UTILIZED ONLY IN CASES WHERE ALL OTHER AVENUES 
FOR CHANGE HAD BEEN INEFFECTIVE, 

As M/>.NY OF YOU ARE ACUTELY AWARE/ LOCAL DISTRICTS HAVE BEEN 

WRESTLING WITH THE IMPLEMENTATION OF AB 293 FOR NEARLY A YEAR 

NOW AND SOME DISTRICTS HAVE/ I UNDERSTAND/ PUT FORTH AMAZINGLY 

COMPETENT FIRST EFFORTS, I AM OFTEN ASKED WHICH PROGRAMS WHICH 

HAVE BEEN ESTABLISHED MOST NEARLY MEET MY INTENTIONS AS THE 

AUTHOR OF AB 293. To MY OWN AMAZEMENT I AM QUITE OFTEN ASKED 
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THIS QUESTION BY THE SAME PEOPLE WHO HAVE QUESTIONED MY 
COMPETENCE AS A LEGISLATOR FOR EVEN INTRODUCING THE BILL IN 
THE FIRST PLACE, 

My RESPONSE IS QUITE SIMPLE. AnY PLAN WHICH MEETS THE 
GUIDELINES OF THE BILL AND HAS GENERAL ACCEPTANCE AMONG TEACHERS/ 
ADMINISTRATORS/ PARENTS/ AND SCHOOL BOARD MEMBERS OF A COMMUNITY 
IS A RESPONSIBLE PLAN. 

It mas never my intention to establish any statewide EVALUA- 
TION STANDARDS. I, OF COURSE/ REALIZED THAT DISTRICTS WOULD/ 
where POSSIBLE/ SHARE APPROACHES TO THE PROBLEM AND UTILIZE 
THE BEST INFORMATION AVAILABLE TO ADAPT EVALUATION PROCEDURES 
TO MEET THE NEEDS OF ALL CHILDREN IN THEIR DISTRICTS, 

But AB 293 is an attempt to accomplish the establishment of a 

FLEXIBLE MECHANISM FOR SETTING STANDARDS AT THE LOCAL LEVEL . 
I MUST EMPHASIZE THAT STANDARDS REQUIRED BY THIS LAW DO NOT 
REQUIRE A SINGLE CRITERIA, ThE LAW DOES NOT REQUIRE THAT ALL 
CHILDREN LEARN AT THE SAME RATE. ThE LAW DOES NOT REQUIRE 
THAT ALL TEACHERS NOR ADMINISTRATORS THINK ALIKE OR ACT ALIKE, 

AB 293 DOES NOT TELL LOCAL DISTRICTS HOW TO EVALUATE— IT 
SIMPLY TELLS DISTRICTS THAT THEY MUST, FOR EXAMPLE/ DESPITE 
PERSISTENT RUMORS TO THE CONTRARY/ THE TERMS "TESTING" OR "MEASURE- 
MENT" WILL NOT BE FOUND ANYWHERE IN AB 293, 

I PERSONALLY BELIEVE THAT TESTING DEVICES WILL AND SHOULD 
BE USED AS A PART OF THE EVALUATION PROCESS/ BUT THE EXTENT OF 
THE USAGE IS UP TO THE LOCAL SCHOOL BOARD IN CONSULTATION 
WITH AFFECTED CLASSES OF CERTIFICATED EMPLOYEES, I HAVE 
SUPPORTED ONLY ONE AMENDMENT TO AB 293 SINCE ITS ENACTMENT/ AND 

THAT WAS TO SPECIFICALLY STATE THAT NORMS ESTABLISHED FOR 
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STANDARDIZED TESTS WOULD NOT BE UTILIZED FOR TEACHER EVALUATION. 
I BELIEVE THAT WE MUST LOOK AT INDIVIDUAL STUDENT PROGRESS 
RATHER THAN SIMPLY THE NORM OF ANY PARTICULAR CLASS, 

I MUST NOTE THAT IMPLEMENTING AN EVALUATION SYSTEM IS NOT 
EASY; NO TASK OF CHANGE IS EVER EASY. AnD I AM NOT SO NAIVE 
TO THINK THAT SYSTEMS DEVELOPED TODAY WILL NOT BE REVISED FOR 
THE BETTER TOMORROW. BUT THE TASK OF EVALUATING PROFESSIONAL 
QUALITY IN AND OF ITSELF WILL HAVE A POSITIVE EFFECT ON THE 
CHILDREN OF THIS STATE. ThE PROCESS OF EVALUATION IS ONGOING 
AND MUST ADAPT TO THE NEEDS OF A RAPIDLY CHANGING SOCIETY. I 
WOULD BE DISAPPOINTED IF ANY SCHOOL DISTRICT ADOPTED AN EVALUATION 
SYSTEM FOR 1972-73 AND DID NOT CHANGE IT IN 1973-7^. 

But GIVEN ALL THIS/ WE STILL MUST GET ON WITH THE TASK, 
I HAVE BEEN HEARTENED TO RECEIVE COPIES OF MANY PLANS ADOPTED 
BY LOCAL DISTRICTS WHICH SEEM TO MEET THE POSITIVE APPROACH OF 
THE LAW. AlAMEDA/ SoUTH WhITTIER/ Mt. DiABLO UNIFIED/ LaWNDALE/ 
AND OTHERS TAKEN MADE POSITIVE STEPS ALREADY. I AM SURE THAT ALL 
DISTRICTS IN TIME WILL MEET THIS PROBLEM IN A POSITIVE MANNER. 

NoW/ ALL OF THIS MAY GIVE YOU SOME IDEA OF JUST WHAT WAS 
INTENDED WITH THE PASSAGE OF AB 293/ BUT I WOULD NOT BE CANDID 
IF I DID NOT DWELL ON A MORE SERIOUS PROBLEM WHICH MOTIVATED ME 
TO PERSIST IN THIS FIELD AND WHICH NOW/ MORE THAN EVER/ APPEARS 
TO BE WIDESPREAD IN THE FIELD OF EDUCATION. 

In a RECENT ARTICLE IN THE LOS. ^ELES. TiMES/ SiDNEY SlOMICH 
TELLS A story: "OnE NIGHT A MAN DREAMT THAT A MONSTER WAS ON 
HIS CHEST CHOKING HIM/ TRYING TO KILL HIM. ThE MAN WOKE IN 

TERROR AND SAW THE MONSTER ABOVE HIM. 'WhAT IS GOING TO HAPPEN 
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TO ME?' THE MAN CRIED. "DoN.'t ASK ME/" REPLIED THE MONSTER, 

"It's your dream," 

i think this story applies to much of what is wrong with 
public education today. professional educators are quick to 
see a threat to their existence in any discussion of educational 

CHANGE, 

I HAVE RECEIVED MASSIVE NUMBERS OF LETTERS ON AB 293 
SINCE ITS ENACTMENT. ThE TONE OF A MAJORITY OF THESE LETTERS 
IS HOSTILE TO ANY FORM OF EVALUATION, ThE UNDERLYING CURRENT 
OF SUCH LETTERS IMPLIED THAT PUBLIC SCRUTINY OF EDUCATION WAS 
AKIN TO SACRILEGE. 

In many AREAS OF EDUCATION IT IS AS THOUGH THERE HAS BEEN 

a moratorium declared on common sense, 

For many years parents have retreated before the experts 

TO THE DEGREE OF ABSURDITY. AnD THE EXPERTS HAVE BEEN 

increasingly unready to openly bring their plans before tl'e 
public for scrutiny. 

Nowhere is this more obvious than in the field of tenure, 
AB 293 did not eliminate tenure— it simply put the light of law 
on the activities of publicly employed school personnel. 
Fanatical defense of lackadaisical performance by a few certificated 
employees has not helped the true professionals but has merely 
served to lower the standards and sully the reputation of all 
educators . 

But the public has finally realized that the monster of 
Sidney Slomich was in its own dream and that in the final 
analysis it could control the outcome. Contrary to the attitude 
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OF SOME IN OUR SOCIETY/ THE PUBLIC SECTOR— THOSE WHO ARE ULTIMATELY 
RESPONSIBLE FOR THE SCHOOLS— CAN AND WILL WITHDRAW SUPPORT IF 
IT IS NOT REASSURED THAT PROFESSIONAL QUALITY IS THE ORDER OF 
THE DAY, 

There is a confidence gap. And if confidence in our schools 

IS not INCREASED/ IT WILL BE THE CHILDREN WHO CONTINUE TO SUFFER. 

And ONE FINAL WORD ABOUT EDUCATION, I AM AWARE OF THE 
FACT THAT NEW DUTIES AND HIGHER QUALITY CANNOT BE ACHIEVED 
WITHOUT FINANCIAL SUPPORT FOR THE SCHOOLS— WITHOUT/ I MIGHT 
ADD/ THE CREATION OF A SANE SYSTEM OF SCHOOL FINANCE IN THIS 
STATE. But THE PUBLIC WILL NOT STRONGLY SUPPORT INCREASINGLY 

higher levels of school support without being convinced that 
increased quality will result from that financial commitment. 

i believe that the manner in which educators and the public 
respond to the challenge of implementing ab 293 will have much 
to do with increased financial support for the schools. 

The good faith and responsibility with which the teaching 
profession meets this new law will have much to do with public 
ATTITUDE. Tenure reform is an idea whose time has come/ and 
cooperation by all certificated employees and school districts 
in implementing the new law is the only insurance they have 
that tenure will be maintained. 

Cicero told the unvarnished truth in saying that those who 

HAVE NO knowledge OF WHAT HAS GONE BEFORE MUST FOREVER REMAIN 

children. 

What has gone before in teacher tenure has not been success- 
ful—it MUST NOT BE REPEATED. It HAS JEOPARDIZED THE PRODUCT. 

A QUALITY PRODUCT— A YOUNG PERSON PREPARED FOR LI VING— SHOULD 
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BE OUR COMMON GOAL IN THE SCHOOLS, ThIS CAN ONLY BE ACHIEVED 

with diligence/ dedication/ and common sense on the part of 
all concerned, 

It's up to all of us. 

Let's get on with it! 



# # 
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EVALUATION: SACRED AND PROFANE 
(Abstract) 

Arthur P • Coladarci 



(Arthur Coladarci is Dean of the School 
of Education at Stanford University and 
Professor of Education and of Psychology. 
He received his B.A. degree from Western 
Connecticut College and both his M.A. and 
Ph.D. from Yale University.) 



The level of our technical and conceptual understanding 
of evaluational processes has risen emphatically in the past quarter 
century. The application of this knowledge in our educational insti- 
tutions, however, continues to be relatively primative, u^icritical 
and ineffective. 

In these ccmmeiits, I wish to suggest some distinctions 
that may be heuristic in this connection and a danger that is avoid- 
able. 

Three Faces of Evaluation 

It is useful, I believe, to realize that "evaluation" 
is not a process, similar in form, dynamics and logic wherever it 
is applied. Evaluation may differ considerably across differing 
evaluative purposes and functions. Three rather distinct purposes 
are involved and they differ in process and consequences (I "piggy- 
back" here on a distinction offered by Lee Cronbach almost a decade 
ago) : 

(1) Evaluation for decisions about administrative, regulatory 
and procedures . The purpose here is that of judging the 
adequacy and efficiency of the administrative organization, 
operations and procedured rules of the institution, with 
emphasis on structure rather than functions of the insti- 
tution. The possible concerns are infinite in number: the 
degree to which the institution's organization permits change, 
the fidelity of budget administration, the availability of 
a place to which a teacher may retire to contemplate his 
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divine essence or (less divinely) repair a faculty zipper. 



(2) Evaluation for decisions about students . The purpose here 

is to make and remake decisions on selection, ^^needs/^ status, 
placement, career, assignment, etc. In this function in- 
dividual differences are central and relevant, in contrast 
to (1) above. 

(3) Evaluation for decisions about the institutional program , 
method, content and organization of learning . The reference 
here is clearly to the primary raison d^ ^tre of an educational 
institution and, accordingly, one would expect this 
evaluational function to be dominant in attention and com- 
petence. However, I am constrained to the conclusion that, 
confounding this function with the previous two, we perform 

it badly in the everyday work of the school. The controlling 
concern is (or, rather, should be) assessing the effectiveness 
of instructional techniques, organization of instruction and 
materials of instruction. (Compare the modal use of tests 
to determine whether a pupil should be consigned to hell, 
purgatory or heaven.) The conception of -^teaching as 
hypothesis -making, " which I introduced many years ago, con- 
tinues to be helpful (unlike most of my earlier ideas!) at 
this juncture. Consider, however and unhappily, how little 
of the enormous "testing and evaluating" business in our 
schools is explicitly or even accidentally addressed to the 
assessment and improvement of their raison-di^ 6ntre function 

teaching . At the system level , each year, batteries of 
tests are administered, scored, summarized and stored with 
only rare reference to their implications for system-improve- 
ment. Also, at the system-level, and assuming (awkwardly) 
that the tests are relevant to instructional purposes, con- 
sider the continuing inefficiency in time an^d costs in 
administering all tests to all pupils^ when the data needed 
for system decision can be obtained by sampling both pupils 
and test subsets. At the teacher-spi cific level (to me 
the most critical and productive) , consider how infrequently 
data are produced tliat are relevant in kind and immediacy^ 
of -availability to assessing and modifying instructional 
decisions. (I suspect that the savings in time and funds 
resulting from sampling pupils and tests in system-wide 
testing would go a long way toward making it possible for 
teachers to learn hovj to dt^velop and ase measurement-evalua- 
tion procedures immediately appropriate to the continuing 
teacher-specif ic task . ) 

The third evaluation function, on which I have just commented 
on has both summative and formative components, to use the language 
of Scriven in a now classic presentation. It appears 

to me that while the summative mode will and should be applied in 
making decisions about whether an institutional procedure should be 
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^^exported*' and in periodic reporting to the schools^ clients, we must 
direct priority attention to the application of formative evaluation 
procedures in the daily professional life of the teacher and adminis- 
trator. It is probable, however, that a massive in-service education 
program is needed. The practicing professional educator must learn 
that formative evaluation is neither ''untidy" or unscientific; if 
done conscionably and well; it can be disciplined and rigorous, 
albiet not as unambiguous as summative evaluation techniques. While 
summative evaluation may be conceptualized in many ways, I like the 
following formulation (with apologies to Stake) : 

(a) The continual definition of the purposes and rationale for 
instruction. This is needed also to locate all of the 
reference groups that will be affected by the instruction 
and must be involved. 

(b) Tlic continual identification of '^antecedent data'^ all 
conditions existing priui.- Lo iiit> Liac Liun; "entry behaviors," 
in parallel recent language. 

(c) Tlie continuous identification of "transactional data." The 
instructional process is rhought of as comprising countless 
transactions among pupils, materials, teaclierr. , etc. 

(d) Tlie continuou:i identification of "outcome data." Here we 
refer to all of the data bearing on the intentions of 
instruction. 

The formative task, inherently complex but not 
impossible, is constantly to look for congruencies between trans- 
actions and outcomes, remedying as w<? go. Done well, instruction 
is never the same; i.e. , it is always forming and reforming. 

The Danger of Over s inijjlifieat ion 

I take the liberty of offering a long quotation from 
a statement by Stake earlier this year; it cannot be improved upon, 
in my view: 

The wfiole cloth is a grand accumulation of intense 
transactions and outcomes. The teachers intend to 
deliver on many promises and to take advantage of 
many targets of opportunity in the usual scliool 
setting. Cach child brings his own complex con- 
victions, misunderstandings, and propensities, and 
he takes away some of these and others as well. 
Lach claL jroom is a comnunity with rules and 
stresses , competition, compassion . Yesterday^ s 
subgroups are not tomorrow^ s. Things are learned, 
unlearned, releai*Jied, much as shoelaces are knotted, 
untied, broken, retied. An educational program 
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has couiitloss oD joctivos. many of them dormant 
until ^ crisis arises. The priorities vary over 
tlfi^j, fi'orTi person to person, and no statement of 
program objectives e\(»r devised has come close to 
reprcs :nt injj, the real world intents of people 
involved in an t}ducationaL program. This is the 
whole clotli of an educational piK)^^:^am. 

I sliouldn't imply that we Ccin*t get reasonable 
consensus as to what the priority objectives are, 
but that's consensus. Tlie unspoken objectives 

such as safety in the classroom, sharing of 
work responsibilities, developing a sense of humor, 
a respect for rules, a tolerance of ambiguity, and 
so on are left to take care of themselves, at 
least until a crisis arises. Then these objectives 
may preempt all others. One can get a simple consensus 
for a list ot top objectives as long as no one takes 
the consensus too seriously. Children can still 
got more than a primitive education, the primitive 
education tliat the consensus statement describes. 
A 'jcutury ago, the Swiss historian, Jacob Burckhardt, 
otateci that the essence of tyranny was the denial 

I oiuplf xiry , and he forsaw tliat ours would be the 
a^T'e oi' tlic greit simplifiers. To quote from Patrick 
Moynihari on iUirkhardt, MIo was right, this is the 
single greatest temptation of tiie times, it^s the 
great corrupter, and it niust be resisted with 
purpose and enei'gy. * Consensus is a great simpli- 
lier, and so is tlieory: statistical processes are 
simplifiers, test scores are simplifiers, s.imple 
representations of the complex. These simplifiers 
help us by reducing ttie complex phenomena to some- 
thing within oui* puwer of comprehension, but they 
also mislead us, by saying that education is much 
less than it really is. 

Wc work day by day witli simplifications, with 
statements of objectives, witli central ^.enuencies, 
wi til criteria tests, and we become transfixed by 
them, losing our awareness of the fundanental 
activities of teacliing and learning. Vve do it 
to ourselves and we do it to our audierces, that 
is, the peopl(j for wliom we are articulrting the 
program evaluation. Evaluator? should be helping 
people keep in touch with the reality of instruc- 
tion, but our scrapbooks are full of enlargements 
of enlargements. 

Th.erc is abundant evidence that the danger of over- 
simplification in evaluation is real. It is illustrated unhappily 
in the naive and distorting reductionism that occurs in jumping 
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from definitions of purpose to operations of measurement: a current 
emphatic example is found in school system acrobatics in attempting 
to comply with the requirements of the Stull legislation. 



SYMPOSIUM I 



TEACHER EVALUATION THROUGH MEASUREMENT 
OF STUDENT ACHIEVL>n:NT 



Chalnr.an: Dr. W. James Pophain 

Professor of Education 

University of California, Los Angeles 



Participants: John D. McNeil 

Professor of Education 

University of California, Los Angeles 
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TEACHER EVALUATION THROUCT STUDENT ACHIEVEMENT 



John D. McNeil 



(John D. McNeil Is Professor of Ec'ucatlon at the 
University of California, Los Angeles. He 
received his B.A. and M.A. degree from San Diego 
State College and Doctorate degree from Teachers 
College » Columbia University.) 



*'By th^ work one knows the vorkioan.'' The succcess of the 
teacher Is best shown In the progress of his pupils. Tiiose evaluating 
teachers have» however, seldcn followed this maxim. Judgment of the teacher's 
classroom processes » general background » characteristics » personal conduct in 
the school and comnunlty has dominated the realm of teacher assessment. Now» 
a number of circumstances are making It necessary for those who would Judge 
teachers to collect evidence regarding the teacher's ability to effect pre- 
specified changes In pupils. There Is, for Instance, civil rights legislation 
demanding that decisions, such as teacher selection, rest upon data directly 
related to work with pupils, not upon the educational and cultural accom- 
plishments of the teacher hlmselt. The desire of legislators, administrators, 
and leaders of teachers' «)rganl2atlons to restore public confidence In 
schools Is another factor In motivating measurement of outcomes. The public 
wants assurance that children are protected from Incompetent teachers. 
Evidence collected systematically regarding comparative standing of teachers 
In terms of student achleveraent Is believed to be more reasturlng than tenure 
(longevity) and licensing (courses completed). This Is not to say that the 
only reason for evaluating teachers by results Is so the unfit can be weeded 
out. On the contrary, the number of teachers who may be fired for falling 
to advance the learning of pupils Is Insignificant In comparison with the 
number of those being evaluated so that they can Improve. 

Reservation In accepting student achievement as the principal 
criterion of effectiveness centers chiefly on concerns about the adequacy of 
measures for assessing a wide range of outcomes and failure to account for 
Instructional variables that the teacher does not control. Are these pre- 
occupations valld^ Is educational technology answering them? Let's see: 

1. Criticism that measures of pupil progress are Insensitive , Standardized 
tests have been the traditional Instruments for measuring the effects of 
Instruction. These tests are frequently vulnerable to the charges that they 
are not likely to contain both the breadth and depth of content coverage 
necessary to make a detailed assessment. Further, the results of such tests 
iar the American's conscience about equality by the fact that the tests are 
designed to give a set of scores which will produce a ^'normal*' distribution 
dooming half of a class or half a school to below average. 

Response . There has been an Increase lu availability of criterion-referenced 
tests which focus on many specific objectives, such as critical skills In 
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learning to read, which nearly all pupils are expected to master. Whereas p 
standardized test nay rieasure, say, five objectives for a course of study, 
a set of criterion-referenced tests permits the neasurement of five tiines 
that number of objectives, some of which rlpht be prerequisite to more encom- 
passing terminal objectives. With the standardized tests, the teacher pets 
credit only when pupils succeed on the five objectives regardless of the 
number of prerequisite skills effectively taueht. With criterion-referenced 
tests, teachers have h better chance to receive recognition for whatever they 
have taught successfully. 

2 . Cri t icisn that teachers will teach too trivial, easily attained 
objectives . 

Response. The practice cf holding prcins tructional conference helps to 
ensure more warranted intents. At the conference, proposed objectives are 
reviewed by supervisors, colleagues, parents, and pupils. The quality of the 
conference is enhanced by discussing the probability that the learner needs 
the competency specified in the objective, that the objective is teachable, 
an<^ that the objective will be acquired only through efforts of the teacher. 

3. Criticism that collec ting evid ence of pupil jprogress is too costly . 

Response. One hundred years ago inspectors in British schools engaged in 
pupil sampling, examining evfery fifth pupil. This sampling procedure is 
economical, especially when one is measuring attaininent of objectives that 
call for observation of pupils' processes and oral production. Currently, 
sampling procedures also include the sampling of items and tests. With item 
sampling, each student gets a sample test or sample item drawn from a large 
population of tasks that are taught in the course. Different students take 
different tests, thereby, reducing time required for testing. 

A . Assessment of pupj.1 P_^i?j^^ss ^oes not re veal why the t eacher is 
succ eeding or_f ail i_ng . 

Response. More than seventy observation schedules are available for use in 
recording or analyzing a teacher's practices. These schedules can be applied 
to observation of single lessons and units of instruction by supervisors, 
colleagues, pupils, and even the teachers (provided a record of classroom 
interactions has been made) - 

A most recent tool The_T^aclier Jmprovem^^ has been designed to help 

the teacher diagnose teaching strengths and weaknesses and develop Improved 
teadiing strategies. Kits treating different subject natter and kinds of 
teaching tasks offer a range of minilessons including background information 
about the task, pupil posttests, normative data, and specific procedures for 
getting better results should a need for improvement be indicated. 

5. Comparison of teachers on the_ basis of pupil progress is_ unfa^lr in^^s much 
as__the conduit ions ^der which instruction occur are not standardized. 

Resj>onse. Researchers at Instructional Appraisal Services arc now developing 
controlled performance tests that afford all participating teachers equal 
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opportxinity to display their competency to effect changes in learners (both 
achievement and affect). Tlieir performance tests control for teaching tasks, 
instructional material > time for preparation and instruction, and ability and 
predisposition of pupils. 

* Examples of sources for newer instructional technology include: 

1. Criterion-referenced tests 

a. Wisconsin Research and Developncnt Center — The Wisconsin D^^aign 
for Reading — University of Wisconsin, Madison, 53706 

b. Instructional Objectives Exchange — P.O. Box 24095, Los Angeles, 
California 90024 

2. Observation schedules — ^li rrors for Behavior . Research for Better 
Schools, Philadelphia, Pennsylvania 

3. Teaching Improvement Kits — Instructional Appraisal Services, P.O. 
Box 24821, Los Angeles, California 90024 
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SYMPOSIUM II 



STATEWIDE EVALUATION AND TESTING 



Chairman: Dr. Mabel C. Purl 

Director of Research and Evaluation 
Riverside Unified Schools 



Participants: Alex Law, Chief 

Program Evaluation Office 
State Department of Education 

Assemblyman Leroy F. Greene 
3rd Assembly District 
State of California 
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SYMPOSIUM III 



Chairman: 



Participants: 



DISTRICT EVALUATION PROCEDURES 



Dr. Lester W. Ristow 

Assistant Director 

Division of Research and Pupil 

Personnel Services 
Los Angeles County Schools 



J. Alden Vanderpool 
Teacher Education Executive 
California Teachers Association 

Harold W. Stombs 

Consultant to California Teachers Association 
On Employee Evaluation Systems 

Garford G. Gordon 

Research Executive 

California Teachers Association 

Theodore Bass 

Professional Services Executive 
California Teachers Association 




CERTIFICATED EMPLOYEE EVAUIATION PLANS 



J. Alden Vanderpool 
Harold tf. Stonbs 



(J. Alden Vanderpool is Teacher 
Education Executive for the California 
Teachers Association, Burlingaae. He 
received his B.A. degree from Central 
Washington State College » his M.Ed, degree 
from the University of Washington* and his 
Ed.D. degree from Stanford University. 
Harold W. Stombs is Consultant to the CTA 
Evaluation Review Service. He received his 
B.A. and M.A. degrees from San Jose State 
College.) 



In the 1971 session » the California State Legisl^tUire 
passed the Stull*Rodda Professional Competency Act. While retaining 
employee tenure » the Act outlines new procedures for dismissal of 
certificated employees. One of the causes for dismissal is incompe** 
tence» a charge which has rarely been successfully used in diftsalssal 
proceedings in California. 

Section 13407 requires the notice of Incompetency to 
include the evaluation made pursuant to Article 5.5. This article 
stipulates four elements which must be included in the evaluation: 
expected student progress » preserving a suitable learning environment » 
maintaining proper control » and other duties normally required as an 
adjunct to regular asslgnmi^nts. Connon practice has based evaluation 
on presage and process crit^^ria* Evaluation now must also be based 
upon the product criterion^ expected student progress. 

The Stull-Rodda Act requires that these criteria apply 
not only to teachers » but to all certificated employees. The employing 
authority also has a responsibility to provide assistance to any 
employee whose performance is in need of improvement. 
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Most of the evaluation plans %re have examined start fron 
the premise that evaluation should he a positive procedure that should 
result in improvement in the instructional program. This is comnendable 
and may tend to reduce the threat that is Implicit in evaluation. The 
consequences of evaluation should be positive steps to correct deficien* 
cles rather than punishment or dismissal. Whether or not the process 
of evaluation will fulfill expectations for an improved educatioaal 
product depends on the criteria used to describe a good instructional 
program. 

Several problems result from attempts to implement the 
Stull*Rodda Act. One of the moat comnon problems among those we have 
encountered is that some districts are producing evaluation plans that 
suggest that they do not understand the intent of the law. They have 
written a few goals and objectives and plan to continue evaluating as 
they have always done. While their intentions are undoubtedly good, 
years of practice » lack of understanding and expertise » md the monu* 
mental size of the task have combined so that» in our opinion, few 
districts have fully or successfully implemented the Stull*Rodda Act. 
With no proven models to rely upon, this conclusion is not alarming 
because districts cannot be expected to build new evaluation systems 
in ond year. 

A more profound problem, and one that should be of 
concern to researchers, is that we do not really know what specific 
teacher behaviors will result in better learning by the students. 
This may be due in part to the lack of a theory or thc^ories of 
instruction, with the result that there is little systematic applica- 
tion of learning theories to the practical teaching situation. 

Another serious problem is how the evaluative criteria 
should be applied to non- Instructional personnel. The law requires 
that principals and superintendents be evaluated on the basis of 
e^^pected student progress. We are seeking ways to reduce this 
principle to procedures. 

Another intricate problem is institutional evaluation. 
Traditional assessment tends to be blind to the perceptions of those 
employees who are served by others. This is a reflection of the 
differential rate at which communication icoves up or down through the 
structure of an organization. Higher office can become a refuge from 
criticism, with the result that failure of the system often reflects 
most on those who are not primarily responsible* The problem, then 
is to provide for assessment of administrative and service staff in 
important part by instructional staff, and, at the same time, preserve 
and enhance mutual respect and the dignity of higher offices* An 
obvious way to provide for assessment of administrative and service 
personnel is to include provision for incorporating the opinions and 
Judgments of those supervised or served. Furthermore, to preserve and 
enhance mutual respect and the dignity of higher offices, it is 
necessary to ensure that higher positions are staffed by competent 
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individuals and that those individuals remain alert to their reoponsi* 
bilities* It is as&uined that evaluation can contribute to this end. 



A practical probleta that recurs is an evident lack of 
technical adequacy on the part of nost school district personnel in 
writing objectives. Part of establishing standards of expected student 
progress is stating them as student objectives. Such objectives should 
reflect the educational program and be directed tovard the educational 
goals of the district. They should also be suitable for use in 
evaluating employees. The objectives should not be so broad that their 
achievement would not be expected within the evaluation period p usually 
one year. Neither should they be directed at the daily* lesson level 
wherein great catalogs would be required to contain thea and excessive 
energy would be required to monitor them. We also advise against the 
use of standardized test norms in measuring student progress for these 
purposes. 

The following are items that require particular attention 
when analyzing an evaluation system. They are not arranged in any order 
of priority, and the list is not exhaustive. 

General 

The evaluation procedures apply uniformly to all certificated employees. 

All procedures » definitions, objectives, criteria, and measures are 
formally adopted within the evaluation system. 

When required elements are not stated, such as standards of expected 
student progress, then there are procedures for generating such elements. 

When required elements are not included in the evaluation document, 
reference is made to other documents which are readily accessible, i.e.. 
Job descriptions, district goals. 

A distinction is made and there 1& a satisfactory balance between 
presage, process, and product criteria. 

Certificated instructional employees were properly Involved in establish- 
ing the evaluation system. 

The application of standardized norms to student assessment for purposes 
of evaluating employees has been excluded from the system. 

Evaluators are evaluated on the quality of their skilla in assessing 
and evaluating other employees. 

Evaluators are evaluated on the quality and results of assistance 
provided to evaluatees in need of improvement. 

"Riere are provisions to balance the power discrepancy between evaluator 
and evaluatee when arriving at mutual acceptance of standards of 
expected student progress and other elements of evaluation. 

Definitions of terms are clear and accurate. 
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There are fair procedures for resolving differences regarding autual 
agreement to standards of expected student progress and other eleneats 
of evaluation. 

The guidelines make a distinction between evaluation as a process rather 
than an event. 

There are procedures for institutional accountability (reciprocal, 
contextual evaluation). 

Possibilities for ex post facto evaluation are excluded from the system. 
(No statements such as "performs other duties as assigned.'*) 

The guidelines encourage a colleagual rather than an authoritarian 
relationship between administrators, instructional, and service personnel. 

Report forms include only the adopted elements of evaluation* 

Report forms include all the adopted elements of evaluation. 

The use of all report forms is fully explained. 

Final evaluation report forms yield a profile typical of the employee's 
service. 

Subjective criteria are excluded if rating scales are used. 

The assumption is avoided that evaluation, under the Stull*Rodda Act, 
is synonymous with adopting a PPB System or utilizing that format. 

Deadlines and time allotments for assessment are satisfactory. 

Employee evaluation is an Integral part of an overall assessment of the 
educational program, and the results of this assessment are properly 
reported to the governing board* 

There are provisions for Inservice training. 

Th^re is a procedure for an annual overall review of the evaluation 
system. 

Pupil Progress 

Goals related to each certificated employee are clearly stated. 

Pupil-progress objectives are stated for each organizational unit of 
the school system* 

Success criteria statements are adopted for each objective. 

Procedures are stated for assessing pupil status in relation to each 
success criteria. 

A mechanism has been established for assuring valid application of each 
method of assessment of pupil progress. 

There is a list of conditions or circumstances (mitigating in their 
effect) zo be considered in staff evaluation on the basis of pupil 
progress. 
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The list of conditions or circumsCances includes mininum standards vtilch 
should be net before the eaployee can be held responsible for pupil 
progress* 

There are procedures for assessing these conditions or circumstances • 

Criteria are stated for determining adequacy of these conditions or 
circumstances. 

There are allowances for borderline conditions or circumstances where 
minimum standards have been barely met. 

There are procedures for reassessing fundamental standards of pupil 
progress. 

Maintenance of a Suitable Learning Environment and Proper Control 

''Suitable learning environment and proper control" is defined for each 
organizational unit of the school system. 

There are procedures for assessing the suitability of the learning 
environment and control exercised. 

There are definitions of unacceptable maintenance of the learning 
environment and exercise of control. 

There are procedures for review and reassessment of definitions of the 
suitability of the learning environment and of proper control* 

Adjunct D uties 

There are procedures for determining the load represented by adjunct 
duty assignments, 

Tliere is a statement of objectives and success criteria for each adjunct 
duty assignment. 

There are procedures for assessing the employee's performance of his 
adjunct duties. 

There are procedures for review and reassessment of adjunct duty 
assignment. 

Evaluation of adjunct duties is in proper perspective with respect to 
the other elements of the evaluation system. 

Standards have been established in each area of study. 

When predicting student achievement^ baseline data has been^or will be, 
established. 

Objectives are not so broad and general that they are of no practical 
use , 

Objectives are not so narrow that they are trivial or become so numerous 
that they are unaianageable. 

Objectives are technically well constructed. 



Objectives are directed toward, and consistent with, the educational 
goals of the district. 

In statements of objectives, measurement is to be performed using 
criterion-referenced measures rather than standardized test publisher* 
norms* 
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INTRODUCTION 



An evaluation is a judgment applied to a sitxiation or wO a process. 
It is often thought of as being applied to an individual. This, however, 
is a secondary effect, reflecting the evaluation of the process or 
situation for which the individual is responsbible. It involves applica- 
tion of additional considerations, such as the degree to which the 
individual has responsibility for or is able to influence the process 
or situaiticn. 

While evaluations can be niade in many ways, valid ones require 
that the judgments be inade on as accurate and complete data as possible 
about factors having meaningful relationships to the situations and 
processes being evaluated. The meaningfulness of the relationships of 
any factor to a situation or process depends upon the objectives of 
the process or the aspects of the situation x^hich are considered 
significant. 

There are two aspects of the function of educational systems. One 
is the process of education, which is designed to achieve certain 
objectives in modifying the behavior (in the broadest sense) of the 
students. The other is the maintenance of certain levels of physical 
and psychological well-being on the port of students, parents, and 
community during the time students are subject to the process. It 
appears, therefore, that valid educational evaluation requires 
consideration of both situational and process factors; and that judg- 
mental standards should relate both to the objectives of the educational 
process and to the environment in which it is conducted. 



puRPosrs or schools 



G^naral. The specific purposes of a school system and of individual 
units within it- -classes, schools, departments — depend upon the society 
within which it operates. However, certain things are coiranon to all. 
Education is behavior modif ication--in other words, those who have 
completed their schooling or any portion of it should behave differently 
than they would have behaved if they had never been to school. Such 
modification of behavior may be trivial or profound, it may reflect 
conditioning or deep internal changes of perception, philosophy, and 
emotions. However, only l\\e overt behavior changes that result will 
show whether or not an observable change has been vrrought in the 
student. Anybody is at liberty to hypothesize internal changes not 
reflected in outward modifications of behavior; but he has no way to 
show thAt change has in fact occurred in the absence of outward 
manifestations . 
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A so^coTvl purpose of th^ schools Is th« assursption of responsibility 
for children and youth ov«r a significant part of their life spans. 
While it is hoped that activities conducted by the schools during this 
period will contribute to desired behavior niodification; there are 
other objectives t including maintenance of social and physical conditions 
the cotnwunlty considers desirable for its younger members. Whether or 
not meeting the objectives in this area will pro«ote--or even hinder — 
the attainment of objectives in behavior modif ication^ this function 
of the schools is an accepted part of their role In modem society. 

Any evaluation of a school system or of any units of it must 
recognize the dual purposes of schools. Although the underlying 
philosophy and societal demands have rK^t been made explicit » current 
developments in legislative and other arenas toward requiring greater 
^'accountability*' of the schools make the need to r^ognise the dual 
purposes clear. 



InrSTRUCTIOM 



It is true that the ptirpose of instn^tlon is the modification of 
behavior, dut tM^s must be understood in its bro^est sense. Certain 
segments of the school enterprise are considered primarily to be 
engaged in modification of specific behaviors* This is chiefly true 
in the so<^called skill areas. :4owever, modem communities expect much 
broader modifications also — those of the type implied in statements 
such as» '^ability to get and hold a job»** ''inqproved self* image 
and ''appreciation of cultural differences.^' These broader objectives 
clearly imply that instruction must not only result in specific cog* 
nitive and psychomotor behavioral changes; but that It must operate 
so as to maintain a physical and «notional environment satisfactory 
to the community and conducive to achievement of broader objectives. 



SUPPORT SERVICES 



Support services have two broad purposes « One is the support of 
instruction by providing facilities » materials* and professional 
assistance in instructional activities. TTie second is working directly 
with students to supply the personal assistance and the environmental 
conditions expected and required of the schools if their role expects* 
tions are to be met. 
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AD«1N1STKATI0N 



The t^m administration is used rather than ^'school »ana(«Mnt** 
since one maj^ function of administration is to iMdiate betveen the 
coiiii>4anlr/* parents, and instruct icmal and support personnel. The other 
major function is of course manageMnt per se^^-^the securing and 
distribution of supplies, equipment, auxiliary serrices, physical 
facilities, and the conduct of coordinating and liaison activities. 



ASSIGNMENT OF RESPONSIBILITIES 



General. T^ere are many teaching vice-principals, teaching 
principals, and even teaching superintendents. The custom of using 
counselors who teach one or more classes is widespread. Department 
heads or other lower echelon administrative personnel *re rarely 
free from direct instr\iCtional duties. It must, therefore, be recognized 
that no position title can be assumed to convey the full description 
of the functions and responsibilities of an individual occupying the 
position. Valid evaluation of the performance of any individual must 
be based on an accurate description of the responsibilities assigned 
to that individual and not on job titles, or even on broad •category 
statements of job responsibilities — e.g. , a general stat«f^«nt of the 
responsibilities of the position oi teacher does not provide sufficient 
distinction between or definition of the responsibilities of teachers 
of different student groups* grade levels, and subject areas. 

Teaching MmBpofmibilitime. While broad guidelines as to what 
persons in teaching positions are expected to accomplish are desirable* 
they cannot constitute an adequate basis for evaluation. They must be 
suppleTnented by specific statements as to the results expected. This 
applies not only to pupil progress but to managerial functions such as 
provision of proper environmental conditions and exercise of control 
over pupil activities. Uhen the authority or resources needed for any 
function assigned to a teaching p>osition are not under the control of 
the person in that position, the responsibility must be clearly 
assigned to another position. 

Support R€mpon»ihiHti€9. These may be assigned to any position 
in a school district. Some positions may be exclusively responsible 
for supp>orting instruction by working on the psycholo{;ical and social 
problems cf pupils. Others may be responsible only for the furnishing 
of necessary instructional materials. Host will have several responsi- 
bilities--e.g. , a librarian is typically resj>onsible for providing 
library materials, for helping students learn to use the library more 
effectively, and for maintaining order among students using it on an 
individual basis. 



Adriin: ,':trative Fesponcihility . The responsibility of an adminis- 
trative position can be very broad indeed. However, no person filling 
such a position is Superrnan. Broad statements of responsibility, while 
theoretically true, are usually unhelpful unless supplemented by specific 
stateTT.ents of operational responsibilities. For example, a principal 
m.ay be said to be responsible for the educational program of his school. 
Specifically, though, he is responsible for teacher assignments, for 
r:iaintaining liaison between his staff ^nd the central administration, 
for seeing that clerical tasks are performed on time, and for supporting 
the teachers vis-a-vis parents and problem students. 



GOALS AND OBJECTIVES 



Piirpoce of Goals and Objectives . The original goal of American 
schools as stated, was to teach children to read the Bible. In the next 
two centuries, the preparation of persons equipped to be clerks, 
accountants, and shipmasters was added. Later, the imparting of common 
social and political goals and the stimulating of a common patriotism 
were added. These and other goals were not added for schools in all 
par-ts of tlie country, nor were their implications ever analyzed at the 
local level. They were sythesized into such generalities as the "Seven 
Cardinal Principles" at the national, academic level. 

The natural cc;'isequence of this development was the creation 
of widely varying individual and local interpretations of what the goals 
of the schools and of any given school program really were. As a result, 
any school person was forced to enter into a guessing game and to gamble 
that his idea of the goals he was to strive for would be the ones valued 
by whoever evaluated his work. To avoid the chaos caused by this 
situation, which still exists in gireat degree, educators preferred to 
evaluate each other--and be evaluated themselves--according to how they 
acted, how they appeared, and their experience and academic record. 

Research has shown little connection between thutie presage and 
process evaluations and the achievement by pupils of ai j agreed upon 
levels of success in any area. Which means that evaluation has ber ■ 
a matter of ^'pleasing the boss" and "keeping your nose clean." T? 
movement towards establishment of objectives which are clearly s< . 
forth before school activities commence is an attempt to assure hat 
personnel will be evaluated according to the results they get 
progressing toward objectives that are truly relevant to the ..urposes 
of the schooJ.s. 
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Objectives vs Goals, There is no precise boundary between goals 
and objectives. In a pure forri] a goal is a stateiuent of a purpose of 
the schools. It gives direction but no specifics. Thus, "to produce 
better citizens," is a pure goal statement. On the other side, a pure 
objective is a statement of a specific observable result to be 
accomplished by a particular educational operation within a specific 
time span. For example, "by the end of the second week of school to 
have every pupil write his name in the correct position on each paper 
he does in class," is a pure objective statement. 

Some theorists hold that a sufficient number of pure objective 
statements in relation to any educational goal will suffice to determine 
the extent to which the goal is being served by the schools. This is 
both theoretically and practically fallacious. Theoretically, because 
it does not provide for the interaction among human experiences in 
developing skills and knowledge which may, and usually does, make the 
whole quite different from the sum of the parts. Practically, because 
any human behavior, even such a clearly defined one as reading, 
involves far more specifics than can be noted and recorded within 
the limits of a school day, a school year, or even the entire school 
career of a student. Problems of record keeping and staff time only 
serve to reinforce the impracticality of the pure objective approach. 

If pure goal statements are too general, and pure objective 
statements are too limited, what is the answer? The answer, of 
course, is some form of intermediate statement; or the limitation of 
objectives to be considered in evaluating a school activity to a 
feasible number, with the hope — unverified by research and contradicted 
by most experience — that pupils will somehow fill in the gaps on 
their own. There is no need to accept the latter suggestion, since 
there can be a co"-.plete range of statements bridging the gaps between 
those that are "pure objective" and those that are "pure goal." 

Because goal statements have typically been too vague for any 
progress toward the goal to be monitored while instruction is occurring, 
and because pure objective statements provide for definitive determin- 
ation of whether or not a specific instructional program has attained 
the objective stated, many groups and individuals have chosen to use 
the latter in evaluation. Unfortunately, many have overlooked or for- 
gotten the limitations of specific objective statements and have denied 
that there can be intermediates between these and general goal state- 
ments. This is not true. In the present state of education only a 
few specific objective statements can be genuinely useful in 
evaluation. Intermediate statements are a necessity. 
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Behavioral Objectives » The well-publicized and widely disseminated 
positions of those espousing specific goal statements have led to ei 
serious misunderstanding among many educators about the nature of 
objectives. While it is probably true that educational experiences 
modify the internal psychologies of individuals, even though there is 
no external manifestation of change, the only way data can be collected 
for assessing the effects of educational progi^ams is by observation of 
the behavior of those participating in them. This behavior -may be 
Tnanifest through any type of human activity. Saying certain things 
in response to a verbal stimulus known as a question, is a form of 
behavior. So is making marks in specific places on a form in response 
to a visual stimulus. So is spontaneous reading of a book, or 
x^ef raining from attacking a smaller person. 

It is a fact, then, that all statements of objectives which will 
contribute to the collection of data relevant to how an educational 
program is succeeding must be behavioral. The behavior may be such 
that changes, if any, will be very subtle; or it may be so simple and 
overt that change, or lack thereof, is easily determined. The fact 
that certain psychologists, and promoters of instructional systems, 
have attempted to re-define behavior to include only the latter type, 
is to be regretted and do«33 not alter this fact. 



DEFINITION OF EVALUATION 



Evaluation is a judgment. If it is based on data relevant to the 
purpose of the activity being evaluated, it is good evaluation; if it 
is not based on data, or is based on irrelevant data, it is bad 
evaluation. Evaluation is also a judgment based on criteria for 
satisfaction. If the criteria are realistic, it can be good evaluation; 
if they are not realistic in terms of the situation being evaluated, 
it will be bad evaluation. If either the purpose of the activity or 
the criteria are changed in the course of the activity, it will be 
ex post facto evaluation. Insofar as programs are concerned, being 
ex post facto does not automatically make evaluation bad. However, 
if the evaluation is to be used to determine the adequacy of the 
performance of an individual working in the program or responsible 
for it, then ex post facto evaluation cannot be validly applied, and 
its use is an obvious denial of due process to the individuals involved. 
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REQUIREMENTS FOR EVALUATION 



Objectives. The purposes of an activity must be determined before 
any judgment can be made as to whether or not, or to what degree, the 
activity is succeeding. The essential requirement of an objective is 
that it is possible to determine by some type of observation whether 
or net it has been achieved. It is an improvement if it can be made 
possible to observe partial achievement or "near misses," However, 
quantification of degree of achievement of an objective, or of the 
wideness of the "misses," is not essential, A false appearance of 
objectivity can be given by making statements such as, "By the end of 
the semester 80% of the students will,.," The decision that 80% is 
the objective and that the particular dimension is relevant, are purely 
judgmental. The use of specific numbers does nothing to make such 
an objective more useful. This type of quantification is usually more 
harmful than helpful. 

Dimensions* Very simple objectives call for collection of only 
one kind of data. For example, if the objective were, "to teach pupils 
how to get the basketball through the i.oop," the only data to be collected 
is the number of times the ball goes through. This can be refined by 
recording also the number of misses and mathematical manipulation can 
take place, such as calculating percentage of tries resulting in success, 
variation among students in percentage of success, and so on. 

Most objectives, however, will require collection of more than one 
type of data. For example, if the objective were to "teach pupils how 
to score in a basketball game," we would have to add several dimensions 
to the one of getting the ball through the hoop. Perhaps one dimension 
would concern the number of fouls committed, another might concern 
judgment in passing, another might concern ability to judge when to try 
for a basket, and so on, 

A dimension is an activity or a condition, the quality of which 
determines whether or not an objective has been met. Usually it also 
determines the degree to which it has been partially met or the closeness 
of "misses" for those objectives that are not absolute, (An absolute 
objective is one that is either met or not, and concetming which there 
is no interest in how close the activity came to meeting it, if it were 
no t me t , ) 

In addition to being logically related to the objective, a dimension 
must meet the test of practicality. It may be impossible to collect 
data in a given dimension. For example, an objective might be to 
"increase student appreciation for American literature," Several dimen^ 
sions would relc^te to this objective; but the one that might first come 
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to mind, "how the students^ appreciation actually changed during the 
course," is one on which no data can be collected; at least not until 
mind -reading techniques are perfected. Instead, dimensions from whic'^ 
inferences about appreciation of American literature might be drawn 
must be substituted — e»g», books read, extent of participation in 
discussion on American literature, opinions expressed, and so on. 

Practicality also includes the effort that must be devoted to data 
gathering and recording. There is a limit both to teacher and student 
time and, in most cases, to the clerical and other assistance available. 
Obviously a dimension that greatly escalates the time and resources 
required for data collection and recording should be considered only 
in special cases. 

Criteria. The collection of data for dimensions relevant to an 
objective is not evaluation. The summation of the data can be termed 
an assessment of the results of an educational program relative to 
the objective. Whether the results are to be judged as satisfactory, 
as unsatisfactory, or as partially satisfactory, depends on the evalu- 
ative criteria chosen. 

Criteria must be unambiguous and unequivocal. They need not be 
numerical. In fact, the use of numerical criteria may be harmful 
because they give a false appearance of objectivity to a political 
and subjective process. Some criteria are clearly indicated by the 
objectives to which the dimensions relate. For example, if it be 
accepted that one objective of driver training is to enable students 
legally to drive motor vehicles, then the criterion for success is 
clear. The objective has been met for any student successfully securing 
a licenre and not for one who does not get licensed. 

However, most criteria involve some type of quantitative state- 
ment, such as, "80% of the students will...," "the student will achieve 
a score of 10...," "the student will have a markedly increased ability 
to..." The last is a quantitative criterion even though numbers are 
not used. T])e problem in all these cases, is to arrive at some type 
of consensus as to what degree of success is acceptable. In the 
numerical cases, these numbers are simply guesswork or ar^ judgments 
based on experience or are compromises of several such judgments 
and guesses. In the last example, the definition of, "marked increase" 
or the judgment that the change is a "marked increase" is likewise 
found to be either a guess, a judgment based on experience (educated 
guess), or a combination of several of these. 
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Any process whereby guesses^ opinion, and judgments are coTripronised 
and pooled is a political process. However, in many educational 
situations, the process is more overtly political with staff politics, 
governing board politics, and comriunity politics all entering into the 
establishment of criteria. 

Validity of Evali'iation. The validity of an evaluation is not the 
sum of the validity of its components, but the product of them and 
some extraneous factors. For an evaluation process to be valid, it 
must be based on valid objectives. It m.ust be based on data collected 
in dimensions relevant to these objectives. The evaluation criteria 
must be realistic, related to data, and determined before the educational 
activity begins. 

Extraneous factors apply rnore to the conduct of educational 
activities by individuals and small groups. The assessment aspects 
of evaluation may be applied in any circumstances, but criteria for 
satisfaction and di satisfaction relating to the specific performance 
of the programs and of individuals conducting them, cannot be derived 
for dimensions over which the responsible organizations or individuals 
have no control. In other words, no success criteria can be validly 
used in evaluating any educator or educational function on the basis 
of data in a dimension over which he or it doe., not have control. 



EVALUATION OF PERSONNEL 



Evaluation of Positions, The requirements for valid evaluation 
of educational programs have been discussed above. If a particular 
position carries direct responsibility for the progr-am and carries 
no other responsibility, evaluation of the program is evaluation of 
the functioning of that position. Such situations are rare, if they 
exist at all* They may be approximated in large educational enterprises 
where marked specialization is possible. 

Typically, educational positions have responsibilities for at least 
some aspects of several programs. Teachers, even those with the 
most academic assignments, are charged with advising students from 
time lo time, with ordering or advising on the ordering of supplies 
and equipment, and with helping determine general school policy on the 
application of district policies to such matters as discipline. 
Principals are usually charged with curriculum coordination and 
personnel administration in their schools. They typically have 
responsibilities in other fields such as community relations. 
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coordination with other schools, discipline, and advising the centra} 
administration* Central administrators frequently wear so many hats that 
central office responsibility for various programs is diffuse. 

f^oh Pefi-nition^, As stated in the section on the validity of 
evaluation, valid evaluation must be based on valid objectives. Insofar 
as the entire school system is concerned, it may be held that the 
objectives adopted by tV\e community through the governing board are, 
if properly stated, automatically valid. Be this as it may, objectives 
for various positions are valid onlv if they clesrlv relate to the 
responsibilities assigned to that position. As a reductio ad ahL^iWiJuriy 
it is clear that an objective pertaining to arithmetic computational 
skills would be invalid for the evaluation of a teacher of English 
literature. Unfortunately, the relevance of objectives to the evaluation 
of specific positions is not always so clear. 

The general rule for determining the validity of objectives for 
evaluation of the functioning of a specific position is simple. It is, 
"Is there a clear, recognized and formal assignment of respons U^l Ht^ 
for each aspect of the program to this position?^' If the aspect is com- 
plex, is there a clear determination of which subdivisions of it belong 
to this position and which do not? The functioning of the position can 
only be evaluated when this question has been unequivocally answered for 
all program elements. Naturally, in most cases the answer for some 
program elements will be **no.** Objectives for these elements will, 
then, have no relevancy for evaluation of the functioning of thepsition. 

Personnel Evaluation. Evaluation of the functioning of <l position 
is necessary for the evaluation of personnel assigned to the position. 
But, to use a mathematical metaphor, it is not "suf f icient . First, the 
official assignment of personnel must clearly state that this position 
is one to which the individual is assigned. In the cane of full-time 
positions this may be simple, but there are many positions manned by 
one or more part-time persons. This is frequently true in counselling 
and consultant positions. 

Second, and of vital importance, no individual can be validly 
evaluated in a dimension over which he has no control- Teachers cannot 
b-2 evaluated on the attainment of objectives where text and other 
materials are required for instruction, if they have no control over the 
provision of these materials. Teachers cannot be evaluated on an 
objective relating to student social behavior if they have no control 
of disciplinary processes, interrelations with parents, and other ele^ 
ments relevant to such an objective. Similarly, principals cannot be 
evaluated on the basis of their success in getting supplies to teachers 
if they have no control over the allotment and distribution of supplies. 
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Evaluation. To fulri to b« b«Md oci diM process, rvaliMtioa 
r€^ulr«s €st«bllshMnt of so*^* objo€tiv«s specific onough to 

4#fiM dlMfislons on %ihich data can b« gathtrad. Vary narrov goala 
will tanarally not maka poaalbla an avaluAtlon of tKa progras aa a 
%Awla. Nuwrlcal data la not nacaaaary for valid asaaaaaant of aduca* 
tlooal prograaia. Crltarla for auccaaa ara darlv«d froa ntuaaaaat 
inforMd JudfMnta, and co«ipro«ia«. In no caaa can tKay ba conaldared 
to ba abaolutaa. 

Evaluation is tha application of agreed-upon arit€rt4X to the duta 
collected for dimensions relevant to the objectives adopted for the 

eduoational unit being evaluated. 

Job descriptions. To apply tha raaulta of an avaluation to a gi^n 
poaltlon« it la aaaantlal that tKa raaponalbllltiaa of that poaitlon In 
ralatlon to tha objactlvaa ba una^uivocally daflnad. Whara Indlviduala 
holding poaltiona ara to ba avaluatad, tha poaitlon daacriptioo auat ba 
accoaipanlad by an unaqulvocal aaalgwntnt of authority plua an imaqulvocal 
atataaiant of othar paraonnal raaponalbla for thoaa araaa vKara no 
authority la glvan. 

Individual Evaluation. Individual avaluation can validly ba 
accoaipliahad only by rafarrlng tha avaluation of prograM to tha 
poaltiona raaponaibla* and than applying tha raaulta only to thoaa 
factora for iHtilch tha individual haa both reaponaibility and nacaaaary 
authority to act. Authority doaa not only Man authority ovar 
paraonnal, it alao Includaa authority to aacura auppliaa, aodify pro- 
graM, dlaclpllna atudanta, and Initiata joint actlvltiaa with othar 
paraonnal. 
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STRATEGY DECISIONS IN 
EVALUATION PLANNING 



Theodore Bass 



(Theodore Bass is a senior member of tlie consultant staff of 
the California Teachers Association, with office in San Diego. 
His B.S.E. degree and M.S. degree were received from the Univer- 
sity of Arkansas. His Doctorate was completed at the United 
States International University.) 

The mid -point of any log ical problem-solv ing 
process is the selection from alternatives, a modeled structure 
of provisionally projected action, or in brief, a choice of 
strategy. Seiler points up the issue in system analysis: 

Assuming now, that the poss ible combinat ions 
of action choices have been arrayed, all that is left 
before implementation is choice or decision. Though 
less laborious than any other stage of the problem- 
solving process, the decision state may turn out to 
be the most difficult. It involves three activities: 
(1) measuring available choices against goals which 
have been established; (2) assessing the administrator's 
and the organization's capacity to implement the decision; 
and (3) actually making the decision by committing one's 
self to it . 1 

Seiler does not identify this process as strategy, but it is the 
synthesizing action. It is strategy. 

Strate>^y theory provides, now, a taxonomy of 
strategy for the use of the education planner. ^ Application of 
the taxonomy can be proposed for use in prte para t ion to engage 
in evaluation. 

First is timing . The calendar is inexorable. 
The law itself specifies time boundaries. Reason governs the 
remaining time options. The collection of data about pupil 
needs must come at the beginning of the school term, A five- 
year plan to develop a defensible evaluation program can be 
recoitmended. A short term plan to implement the Stull Bill is 
required. 
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Second comes involvement . Fait Accompli pre- 
cludes involvement. Total democracy involves everyone. Repre- 
sentatives can be designated or elected. Again both law and 
reason require involvement. 

Third in order is knowing . Such concepts as 
criterion referenced tests, professional judgment, normative 
evaluation, grade levels, are strategic. 

Gapi> in data are bridged by theorizing . 
Provisional models of the future provide imagined alternatives. 

Spacing considers approach, choice of media, 
rearrangement of environment. 

Finally, spending allocates scarce resources. 

A grand strategy is a combination of all of the 
foregoing. All of us are strategists by description. The use 
of deliberate prescription in strategy selection characterizes 
responsible planning. The complexities of evaluation can be 
reduced by attention to strategy. 



John A. Seller, Systems Analysis in Organizational 
Behavior . Homewood: Richard D. Irwin, Inc. and the Dorsey Press, 
1967. 

2 

Theodore Bass, A Taxonomy of Strategy T'or Use in a 
System Approach to Education . (Unpublished dissertation. United 
States International University, San Diego). 
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S^TIPOSIUM V 



DEVELOPMENT OF INDIVIDUALIZED EVALUATION STANDARDS 



C hi c* i rrr.3r« « 13 r • J c h n C G ov/ 3 n 

Professor of Education 

California State Universi ty , Nor t bridge 



Participants: Dale Burklund 

Director, Guidance and Vocational Education 
Santa Clara County Schools 



Ezra Wyeth 

Professor, Psychological Foundations Department 
California State Universi ty , Nor thridge 




INDIVIDUALIZED EVALUATION STANDARDS 



E. R. Wyeth 

(Ezro Wyeth is Professcr of Psychol ogice 1 
Pounds t i ons , Ce 1 if orn ie Ste te Uni verc i ty , 
Northridge . His doctorate wbs earned at 
the University of California and his 
earlier academic work weti done in Australia.) 

As I write the newspaper 1 ies open near me . Prominent 
on the spoxts page is the week's ranking of college football teems. 
Nebraska is No 1 and UCLA is not even mentioned in the top 20. TVie 
season has yet to begin. 

For days the paper has exuded gloom. The United States 
is no longer No 1 in Olympic competition. It might even withdraw from 
future games. There has been an organized conspiracy and the athletes 
were beaten by politicians and judges not by their opponents. 

Occasionally e voice is raised against this childif^h 
devotion to the cult of superiority - the adult echo of 'my dad is 
better than ^:>urs'-but the voice is ignored or drowned in the hurt 
protests of editors who defend their practices by declaring that they 
print what the public wants . 

So with the connivance of mass media, John Doe can re- 
joice that his team is No 1 and know on Friday the best player in 
Saturday's game. And this is the John Doe who recites the Pledge of 
Allegiance and sings with gusto the National Anthem, thereby attesting 
to his unswerving belief that all men are equal and he knows in his 
heart that for 200 years his land has been free of those stupidities 
about class and rank that plagued those countries from which his an- 
cestors came. And he is a free man. In fact there are some 8 million 
laws on the statute books in his land - if John Tabori and his book 
The Natural Science of Stupidity are to be believed - to ensure that 
he will be free. 

Unlike Tabori, John Doe is a mass of contradictions. 
He has a schizoid personality and it needs the skill of the best of 
psychiatrists to discover the real John. Whether psychiatry can do 
anything about his condition is another matter. 

I'ou and I have seen something of the ree i John in recent 
months in the issue of school busing and the Eagleton matter in which 
John showed his profound belief in the idea that mentally ill people 
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mtm poss#ss«d d«vils mn6 thmr^ is just no wsy in %^ic^ th*y csn r«» 
ccv^r frosi thst. 

I do not kT;^-w wtio sugQ»*t*d th^ topic for ttiis di.^^cuss* 
ion but h^. m\A^< tm • v#ry br«v« Mn and on« rme.dy to sttack Mcred co*nfc 
Th«r« mrm cold hard facts thst must hav# 9i>fan hisi causa to traad 
%Mirily. Ycu saa ha wnts to attack ona of John's daapast and siost 
charishad baliafs th^»t all »an ara unaqual and sociaty will function 
bast whan avary nan knows his placa and aach is accordad tha honors 
that go with his position. His chil dran aust laam this laaaon early 
in life and John will do his bast to ansura that schools and univarsi* 
tias p^^y thair part - not of coursa that thay will offar m^rh oppo«i* 
t ion . 

Let no r>na try to \jRidarstata tha anorwity of tha task 
inv lv«ti in gatting John to giva up his naad to know whara ha stands 
in ralation to othars. Ha has to ba traatad for a long standing ob» 
sassi'^n and his vary way of lifa has to ba thraatanad. 

Thara ara sporadic attawpts to giva up raport o^rds in 
schools. In many oasas tha attampts ara nmt with opposition from 
parents. Thay naad thosa cards badly. 

F^ut taachars oppomm giving up raport cards as wall. 
Thair naads ara thraatanad. Tha taachar - and lat ma includa tha pro. 
fassor hara - holds graat powar in his hands. Ha assigns ^ot gradaa 
but status and this givas him a aanaa of po^rar - alsKsat a divina mmtmm. 
No wondar ha oftans rafars to himsalf as *%ra'. 

yhat wii: thasa p&llcrs of conaarvatism * tha univarsi* 
xi9>^ * 6r if students d*^ not produrra avidanca of states %^an thay 6«^k 
entry? Mind you univarsitias hava not really placed much reliance on 
the evidence and some of the mora notorious ones have shown their be* 
lief thst the evidence was vrong by failing nearly half tha students 
in thair first year. Thay might even have to follow the path of that 
heretical institution in New York that does not aak for ryfiport cards 
and tha like. Or the path of Junior colleges that show little interest 
in where a student stands in ralation to his peara and aomahow have a 
much lower failure rate than the prestigious universities. 

Let me return to where I started. Tha idM that there 
should be individual evaluation - if tranafarred to tha field of sport* 
would require that the athlete measure his parformsnca against %^t ha 
is capable of doing and winning a gold medal or being No 1 should be 
incidental to that. Even in losing there should be graat satisfaction 
if he performed aa well as he could. 

It also asks that millions of avid fans giva up scream* 
ing that they are No 1 in the land and begin aaying sosiathing about 
winning not being %#hat counts but how tha game \mm played. 
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SYMPOSIUM VI 



THE IMPROVEMENT OF LEARNING THROUGH EVALUATION 



Dr. Carmen J, Finley 

Director of Assessment and Principal 

Research Scientis t 
American Institutes of Research 
Palo Alto 



Participants: Eva L. Baker 

Assistant Professor of Education 
University of California, Los Angeles 

Warren Kallenbach 

Professor of Education 

California State University, San Jose 
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IMPROVIIfG DfSTRUCriOIT THROUGH EVALUATIOT: 



PRECEFD OR PERFIDY? 



Eva L. Baker 



(Jhra L. Baker is Assistant Professor of Education 
at the University of California, Los Angeles. 
She received her B.A., MJl. and Doctorate degrees 
from the University of California, Los Angeles.) 



The intention to use evaluation data as the primary source of 
instructional improvement is underscored by the growing prominence, if not 
acceptance, of accountability models. Little specific analysis is evident 
regarding the feasibility of instructional improvement based on typical 
sources of pupil achievement data. The best approximation of data based 
instructional improvement comes from the field of instructional development, 
where large-scale programs have been designed and revised against a continued 
influx of pupil performance data. 

Data Gathering 

After decisions about goals have been made, hopefully employ- 
ing some of the refined needs asse«!?ment procedures presently available, the 
improver of instructional programs must decide upon measures and proced\ares 
for acquiring relevant data. The trade-offs between well known, frequently 
used adiievement testj and newly developed domain-referenced tests must be 
examined. Of particulai* inteT'est beyouu the quality of information each 
test type yields, is the differential cost of materials, not only in initial 
acquisition, but in terms of the training or orientation required of teachers 
expected to employ the Instruments. The devtlopment of methods of regular 
data acquisition and interpretation must be accelerated. Moreover, it is 
clear that whatever the data collection ventvtres adopted, teachers wi2JL need 
to be acclimated to the continued need for instructional hiatus in the name 
of evaluation. 

Criterion Setting 

To decide in advance what constitutes a deficiency in pupil 
performance normally requires a senr'e of history and iniri)ose not consonant 
with the tradition of achievement measurement. ?re-set criteria, while 
satisfying the need for order, may not reflect feasible levels for achieve- 
ment in the EchoolSw The extent to which criterion statements generated for 
instructional purposes resurface as political goals needs to be both 
considered and anticipated in planning. Perhaps an alternative to pre-set 
criterion statements which reflect percentage of achievement figures mi^t 
be normative statements for individual teachers ^r schools. Teachers (or 
schools) would be compfitred against themselves, with the simple dicttmi that 
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significant improvement is required. Another possibility is to require 
instructional improvement of only those schools or in only those subject 
areas irtiere performance is noticeably below state levels. Thus, improvement 
would not be mandated across the board for a1 1 subject areas and all schools 
or teachers. Individual contractual arrangements would be made according 
to differential need. 

Instructional Improvement 

Instructional improvement cycles, even in the presence of 
interpretable data and adequately defined criteria, require a series of 
complex decisions for the teacher. Of necessity the teacher must make a 
judgment regarding >rtiich instructional activities are likely at fault for 
poor pupil performance. Second, the teacher must analyze acceptable alter- 
natives to prior inadequate instruction. Third, the teacher must be able to 
implement suitable alternatives. These steps demand a large training invest- 
ment and are additionally hampered by the lack of hard knowledge in the 
instructional area itself. 

Certain guidelines can be rou^ly delineated for instructional 
improvement strategies, however. They are based on a two-pronged criterion: 
l) to what extent is there work in the research literature which supports 
the value of the interv'-ention and 2) what are the cost considerations of 
implementation. Instructional interventions can be conveniently scaled in 
terms of this dual consideration. Teachers can be tau^t to analyze their 
p/rtivities so that research -based instructional principles are employed. 
While training must precede such analyses, the actual cost is low of imple- 
menting in a classroom principles such as practice, knowledge of resxilts, 
certain motivational interventions, and other instructional principles with 
considerable support in the scholarly literature. The second type of 
instznictional activity change centers about the method of presentation for 
instznictiou. For instance, should phonics or linguistic approaches be 
employed in the teaching of beginning reading? The research-base comparing 
methods is less strong than for instructional variables, for conflicted 
findings is usually the case attributable tn poor experimental design and 
controls. In addition, the cost of renovating instructional programs to 
conform to a particular method is high. 

Only where the method alternative is encompassed in a validated 
instructional program are method alterations likely to pay off in the class- 
room. The reason for such pessimism derives from the single \inquestioned 
finding in educational research: main effects are rare. Instead, methods 
are likel;^*- to be contingently effective and their utility will depend upon 
the natizre of the learner, the style of the teacher and exigencies of the 
instructional setting. 

Thus the flaw in mandated instructional improvement is the 
expectation that knowledge about such contingent relationships is available. 
While the conduct of serious experimental research testing controlled varia- 
tions would seem to be implied, the question of replicable treatment is most 
likely to be \insatisfactorily considered in the domain of instruction. Until 
a sufficient number of validated programs have been developed and implemented 
which include replicable characteristics, the outlook for painless instruc- 
tional improvement is grim. 
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IMPROVING EDUCATION THROUGH EVALUATION 
Or 

WHO'S GOT DESIGNS ON EDUCATION? 

Warren Kallenbach 



(v^arren Kallenbach is Professor 
of Education at the California State University, 
San Jose. He received his A.B, degree at Drury 
College and both his M.A, and Doctorate degrees at 
Stanford University.) 



Improving education through evaluation? Can we improve 
education through anything? If we accept the Coleman/ Jencks thesis 
perhaps we should look for little more in schools than "wh ther the 
teachers and students find it a satisfying place to be."l Jencks found 
little in Coleman's data to encourage him to believe that the schools can 
contribute significantly to even adult well-being. He does feel that, 
school reforms do improve the lives of children, however. 

Why do we want to improve education anyway? 

Because of the Stull act? 

Because of "Ifliat we're s 'posed to do"? 

Because we want to do good? 

Because we have a Title I, II or III grant? 



1 

Christopher Jencks et al . Inequality: A Reassessment 
of the Effect of Family and Schooling in America. New York: Basic 
Books, 1972, passim . 
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Should we be talking about changing behavior instead of 
^ tiip 1" o V i •T"" Tf Sw- ^hcnc!: bchs%'icr. for hcv? lens, in v?hcit 

direction and for what reasons? Does evaluation or the feedback from 
evaluation make a real difference to the evaluatee? In other words can we 
really improve education through research, development and dissemination 
efforts? 



The federal government is getting ready to put some heavy 
money on just this assumption. It's getting ready "to spend billions where 
spending only millions has produced disappointments. "2 xhe National In- 
stitute of Education is gearing up, amidst some HEW strife, in hopes of 
making some if not many significant changes in education. It is assumed 
there will be many approaches proposed to and carried out by and for the 
NIE, Most will surely include systems approaches and use accountability 
procedures- 

What help can we expect from currently available evaluation 
designs? Well, we have just about buried the once the widely-popular do- 
it-yourself model , namely, get some money to try something, tack on some 
easily-available standardized tests or get some equally easily-available 
evaluation expert--af ter we commence our project, of course. Some more 
desirable models are available, specifically, those of Stuff lebeam, Guba, 
EPIC, CSE, or Provus, respectively, to name several of the most productive 
ones. If we accept the thesis that the major purpose of evaluation is to 
provide "useful information for judging decision alternatives, the above 
designs can be most useful to us. 

There are some research and development centers, some 
regional educational laboratories, and some ESEA Title III projects that 
can provide exemplary models for change. There are some others that do not. 
Some programs or projects that have yielded evidence of changes in learner 
behavior are (1) the Hawaii English Project, (2) the Far West Laboratory 
for Educational Research and Development minicourses, (3) the Individualized 
Mathematics System, (4) the University of Wisconsin R and D Center ICD/MUS-E 
Project, and (5) the Southwest Educational Regional Laboratory Reading 
Programs. These programs were carefully designed and carried out. 



James Brann. "NIE: New Life or Research," Saturday Review of Ed- 
ucation * September 16, 1972, Volume IV, p. 43. 

3 

Daniel L. Stufflebeam et al . Educational Evaluation and Decision 
Making. Itasca: F.E. Peacock Publishers, Inc., 197L 
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Their evaluative data indicate positive significant changv^s in learner 
behavior following application and dissemination of the project treatment 
procedures « Keep in mind though that the most widely-disseminated current 
program in education--the School Mathematics Study Group — " never had any 
formal desigri for development and dissemination/* ^ 



What model do we follow for instructional improvement? A 
highly-structured one? A loosely-organized one? The section by Evi* Baker 
offers some explict recommendations on both these and other approaches. 



4 

Personal Comnunication to the Author, September 1972. 
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COMPETENXY BASED CURRICULUM 



Associate Professor 
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Leo P. Ruth 

Supervisor of Teacher Education 

and English 
Department of Education 
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airman: 



Dr. William Watts 




BEHAVIORAL OBJECTIVES 
AND THE DANGERS OF SYSTEMTHINK IN EDUCATION 



Leo Ruth 



(Leo Ruth is Supervisor of Secondary Education 
(English-Reading), School of Education, University 
of California^ Berkeley. He received his B.A. degree 
from Chico State College and his M.A. degree from 
the University of California at Berkeley.) 



A new style of planning and problem sclvirg, heralded as the 
way to salvation from error in managing Big Education, is sweeping the nation, 
Business and governmental agents are pressing "systems analysis" (including 
behavioral objectives) upon educators, using the ^orce of money and legisla- 
tion to win quick, unquestioning commitment to various forms of so-called 
rational decision making, still largely untested in education. (Significant- 
ly, through adoption of ACR No. 98 [the Dunlap-Vasconcel los Resolution on 
PPBS] in Summer 1972, the California Legislature halted the pell-mell rush 
to install PPBS in California until further hearings are concluded on the 
matter.) 

In California, the loss of a humanistic orientation in educa- 
tional planning was forecast as early as I96A in the "Little Report" (THE 
EMERGING REQUIREMENTS FOR EFFECTIVE LEADERSHIP FOR CALIFORNIA EDUCATION): 

We were struck by th* rate at which pedagogy is 
becoming transformed into educational technology. In- 
creasingly, the design of educational programs involves 
an integration of facilities, personnel, and processes. 
Thus pedagogical considerations are affecting the way 
schools arc designed and the way teachers, materials, and 
pupils are "programmed ." (Italics added) (p . Vs) 
Of course, this unwitting (7) failure to distinguish teachers 
from materials as "operating units" within a design restricts the freedom 
and responsibility of the teacher. The "new Utopians" of education in their 
search for planning strategies to shape a world free from human imperfections 
are promoting mechanical planning models which diminish the scope of human 
responsibility within operating structure ostensibly concerned with people, 
but actually concerned with people^s ubs t i tutes - -computer hardware, system 
procedures, programming, automated tasks, operating unit approaches. 

Currently, much educational planning is gripped by a kind of 
"prosaic mentality" wh'ch promotes a dehumanized, "teacher-proof" "learning 
package" and "management by objectives." This "sys temth i nke r" is attracted 
to mechanical processes, and to the facts. Facts are the abstractions 
which can be assigned numbers. Especially "real" in this 198^ world are the 
events that are quantifiable, measurable. The prosaic mind likes things 
sharply defined, praises objectivity, practices method, loses itself in tech- 
nique. Though the sys temth inker uses the language of science, he understands 
little of science, perpetrating fallacies and rigidities of method that real 
scient ists renounce. 

83 



ERIC 



The burden of this paper is to demonstrate the specific dan- 
s of this type of sys tenth i nk In education. Some of the most evident 
gers are these: 

Reality and truth get replaced by systems models, which then assume a 
reality apart from the actual world they purport to reflect. 

Simplifications inherent in even the most sophisticated models have re- 
ductionist effects upon conceptions of knowledge, learning, and the 
learner . 

An image uf man as a reac t i ve be i ng rather than a c rea t i ve being domin- 
ates these mechanistic sys tems mode 1 s . 

Related to this robot image of man is a simplistic conception of teach- 
ing/learning which evaluate^ its "output" using such criteria as the 
following: speed of response; number of errors or trials; number of 
an i ma 1 s (men) respond i ng; e .c , 

Preoccupation with pi ann i ng techn i que fosters acceptance of illusory 
simplicities and generates confidence in mere changes of structure 
as the way to improve operation. 

Techniques of formulating "behavioral" objectives related to system 
goals are specific (and rigid), but they offer no basis for distin- 
guishing meaningful and meaningless substance. Indefinite numbers 
of objectives can be gene ra ted that meet systems goals, but they 
may be trivial, irrelevant, or even destructive of true understand- 
ing in a field of learning. 

Systems procedures inaugurate, strengthen, and perpetuate a strong 
centralizing bias in the allocation of power for decision making. 

Systems procedures may very often serve chiefly political ends of 

reallocation of power under the guise of bringing about "improvement, 
but usual 1 y such procedures perpetuate existing states of knowledge, 
existing values, established rather than emergent situations. 

Systems models in education tend to be efficiency oriented rather than 
human itarian in aim. 

This desire to be efficient, precise, and predictable restricts pro- 
ductiveness and creative, unexpected response. How do outcomes not 
preconce i ved by the systems desig. (Cr take place? 

In view of the lack of any theory to predict the consequences of wide- 
spread application of a PPB System to all sectors of government, or 
of any great body of research, current incautious promotion reflects 
irresponsible endorsement of as yet incompletely evaluated systems 
approaches in education. 

The frequent disallowance of alternatives to systems approaches to or- 
ganization of education constitutes a contradiction of a first prin- 
ciple of systems thinking, evaluation of alternatives. 

The lack of any data base whatsoever for such widespread imposition of 
the PPB System upon education reflects disregard of another basic 
principle of rational decision making, thot of basing analysis, 
planning, and decision making on data. 

The currer.t hard sell of PPBS in education consequently is a matter of 
ideology resting largely on faith. 

The power of these monolithic conceptuol schemes to structure people ^•nd 
events in education reduces individuals to "rul e- f o 1 1 owe rs" and "role 
players" in the system, denying both pupils and teachers freedom to 
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hVALrATlON OF /VDMIMSTRATIVT ^ SUPERVISORY PERSONNEL 



Robert A. Hansen 



(Robert A. Hansen is Administrative Assistant to 
the Superintendent, Fresno Unified School Dist- 
rict in Fresno, California. He received his B.S. 
degree from Drake University and his M.S. degree 
from the University of Southern California.) 

Evaluation of personnel, a*" a process, can only be defended 
if the ultimate goal is to "improve" the evaluatee. The entire concept of 
improvement requires the identification of at least what is better, and 
preferably what is ideal. 

In education, historically we have been guilty of expending 
vast amounts of resources to improve teachers, in most cases without even 
luakiu^ an attempt to ideiitlfy what was a better teacher. We have been 
willing to have extensive inservice training classes; we have been willing 
to provide opportunities for teachers to take special courses at the local 
university; we have had released time, we have had supervisors and co- 
ordinators visit the classrooms; we have given much advice and training; 
bat we have now matured to the extent that we are willing to at least 
identify for good, for bad, for weak, or for strong, what we feel should 
be required for any job. Of course, this is the key to the whole process 
of evaluation-- we can only evaluate a person's effectiveness if we have 
clearly identified what is required of him. 

The simplest form of ^valuation, then, of administrative and 
supervisory personnel, is to consider their capabilities as they relate to 
their job description. ClearL> then it is required, at the very least, to 
have for every pobicion in the tchool district a recently reviewed and 
adopted job description. It can be as detailed or as generalized as the 
district desires it to be; however, it must be recognized that if it is 
going to bt? used for evaluation, then it should be adequately defined and 
explicit enough to provide for a comparison between what is and what should 
be. Just using the job description would be a minimum evaluation. 

The Stull Bill provides all of us with a charge to evaluate 
ef f ec t i '"oness as it relates to our real product, the student. When we 
consider the evaluation of supervisory and administrative personnel in the 
school district, it follcv^s that we should use, rather we must use, the 
minimum requirements of the Stull Bill. Article 5.5, Section 13487 of the 
Education Code says: "The governing board of each school district shall 
develop and adopt specific evaluation and assessment guidelines vrhtch shall 
include but shall not necessarily be limited in content to the following 
elements: (a) The establishment of standards of expected student progress 
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in each area of study and of techniques for the assessment of that progress. 
(>.) A --r -^.-^r-.rr.t certificated personnel competence as it relates to the 
established standards, (c) Assessment of otrier duties normally required to 
be performed by certificated employees as an adjunct to their regular assign- 
ments, (d) The establishment of procedures and techniques for ascertaining 
that the certificated employee is maintaining proper control and is pre- 
serving a suitable learning environment." 

One form that we are trying this year in Fresno with super- 
visory personnel is to use a process which provides that initially, in the 
fall, each evaluatee takes a form and fills in his own job description in a 
manner which could be construed to be behavioral objectives. He also fills 
in his personal goals and objectives professionally for the year, and then, 
in addition to the job description, if there are some unique objectives or 
assignments that he has been given, they are also filled in. This is done 
by the evaluatee. The evaluatee and the evaluator discuSL^ what he has put 
on the form, and the evaluator either agrees, amends, or deletes any of the 
items and so signifies on the form. During that initial interview the de- 
velopment of the objectives might require that the evaluator, during the 
school year, make some observations, in effect monitor some of the activi- 
ties that have been purported to be accomplished. If so, the form provides 
that he vill note when he has made the observations. 

At the close of the school year the two meet again, but prior 
to the meeting the evaluatee does a self-evaluation of each of the items on 
a simple ranking scale. He then brings the form to a meeting with the eval* 
uator. Together they look over the original set of objectives, any moni- 
toring^ that has been done by the evaluator, the se If -evaluat ion of the 
evaluatee, and then the evaluator marks on the fonr, his own evaluation of 
each of the objectives. If any of the objectives require improvement, he 
marks on the form the kind of improvement expected and the kinds of help 
that will be given. The completed form is signed by both parties; the eval- 
uatee retains one, one is placed in his file and one is retained by the 
evaluator. 

Of course, the process of evaluation is as old as time. We 
do it all day, every day. We hire people; we fire people; we reprimand 
people; and we commend people daily based on our subjective evaluation. It 
seems to me the only difference here is that we are ordering our process to 
the extent that each person will clearly ui/derstand under what conditions 
he is being evaluated, what is being required of him, what he will be held 
accountable for, .md then we are providing at loast the opportunity for clear 
discussion of the discrepancy between his demonstrated performance and his 
expected performance. Also, the district is required to help him "close the 
gap/' l<irhtly ust^d, inipl (imcnti ng; the design of the law (Stull Bill) will: 
(a) provide the public with the- knowled^-^e thar. *'eduCo t ion'* is holding 
personnel accountable; (b) assure?, improved performance of all levels of 
supervision: and (c) provide internal security to the individual, knowing 
that his job is defined and he will be helped if needed. 

Now let's maKC. it wcrlc. 
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Berkeley Unified School District 



Participants : Beth Barmack 

Teaching Assistant 

East Campus, Berkeley High School 

Victor Ichioka 

Teaching Assistant 

East Campus, Berkeley High School 
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STUDENTS AND TEACHERS, PARTNERS IN SELF-EVALUATION 



Beth Barmack 

(Teacher, student, poet, leader in women *s group work.) 

Much of this conference is devoted to the **r,ard data** of 
evaluation and testing. Great stock is placed in the vast and re- 
munerative achievement testing enterprise. 

Little weight is given here or is often given to the role of 
stadents and teachers in evaluating themselves, each other, and the 
educational system of which they are the critical parts. We submit 
that great weight should be put upon their self-assessment; for in 
the classroom, the student is the consumer, and the teacher is the 
prime mover. 

Today, public education is in the doldrums, the butt of vast 
criticism. From all quarters come cries for accountability, year's 
growth for a year in school, make 'em learn. Into this caldron is 
thrown the million dollar question of teacher evaluation, as if this 
were a simple matter of numbers, pluses, minuses, statistics, manipu- 
lated with utter sophistication, interpreted with utter naivete. 

Therefore, as a panel, we must ask such critical questions as: 

1. What is being evaluated anyway? 

2. What is to be learned? 

3. Can we make 'em learn? 

4. Make 'em learn for what? 

5. Finally, what about teachers and teaching? 
Students and learning? 

Today, your chairman has brought with him two uniquely equipped 
and successful teachers of the "newer" and "freer" school, who are 
indeed with their students. Both have urgent messages which point up 
the student's role in teacher evaluation, evaluation of the whole 
educational system, not to mention the social and political system 
of which education is an instrument. At the same time they point up 
the teacher's simultaneous role in the whole process of evaluation. 
They focuft on a partnership in self-evaluation. 
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Victor Ichioka 



(Artist , teacher , student) 

I came to this conference mainly to share questions about our 
school system. My main experience with public schools has been going 
through them. I am not a professional, and, perhaps, that may have 
its advantages. At any rate, I am still more a student than a teacher. 
I intend to look at schools from that point of view. 

I think any inquiry into the worth of evaluation finally ends up 
with questions about the worth of the school system itself. I think 
questions like: What is taught? What is learned? Who benefits from 
the schools? These underlie the whole concept of evaluation. These 
and other such questions should be raised and talked about. 

I rb^rik also that any discussion of evaluation has to include a 
look into exactly what controls a student and his family have over his 
educational destiny. What kinds of choices does a student have? Are 
the schools serving the needs of the student and his family? His 
community? Who determines the student's needs? Who determines whether 
these needs are fulfilled? 

I have no answers. But I believe that if we are to investigate 
the value of schooling, we should ask the most basic questions we know 
how. 

Here, then, are the iriakings of lively and deep discussion, calcula 
to raise basic questions on teacher evaluation. 



94 



SYMPOSIUM X 



Chairman: 



Participants : 



COST EFFECTIVENESS 



Dr. Wallace R. Muelder 

Associate Dean 

School of Education 

University of Southern California 



John Stallings, Chairman 

Department oi Educational Acvanistration 

University of Southern California 

Richard B. Horno 
Assistant Superintendent 
Business and Financial Services 
Los Angeles County Schools 




COST EFFECTIVENESS IN EDUCATION 



John W. Stallings 



(John W. Stallings is Professor of Educational 
Administration at the University of Southern 
California, Los Angeles* He received his B.S.^ 
M.S., and Doctorate degrees from the same 
university. ) 



Many an educator^ facing problems old and new, 
must agree often with the advice offered by Machiavelli nearly 
50 years ago, "There is nothing more difficult to take in 
hand, more perilous to conduct, or more uncertain in its 
success, than to take the lead in the introduction of a new 
order of things." A new order of things is needed if educa- 
tors are to respond to the demands of the legislature and of 
taxpayers for accountable programs. And leaders are a neces- 
sity if the California system is to take this new order in 
hand. A new order of things in this instance must include 
a decision making process for public education based on 
analysis of alternative approaches to resource allocation in 
terms of cost effectiveness. This analysis reflects an 
aspiration shared by all educators: the ability to relate 
the outputs of schools — the educational product — with the 
inputs, measured in terms of financial resources* human 
resources, time and material resources. 

Elements of Cost Effectiveness . What are the 
elements necessary for the application of cost effectiveness 
in education: First, it is necessary to find a workable 
definition for cost effectiveness. 

Cost effectiveness analysis was defined in 
the Planning, Programming, Budgeting System Manual for 
California School Districts as a method of determining the 
most efficient mix of activities to achieve a specific 
objective. Total costs are related to effects. Costs are 
measured in dollars, and effectiveness is expressed in terms 
other than dollars. 

Cost effectiveness differs from cost benefit 
analysis. Cost benefit analysis seeks to establish a ratio 
of costs to benefits when both costs and benefits are 
measured in monetary terms. The inherent difficulty of 
placing a dollar value on educational outcomes makes cost 
benefit analysis extremely difficult and useless for practical 
purposes in school administration. Cost effectiveness 
analysis, however, does offer some hope for potential vdue 
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in helping the decision maker arrive at the best answer when 
various alternatives are under consideration. 

One of the initial requirements for gaining 
significant cost effective comparisons is to hold constant 
either the objectives or the cost. If you were given the 
task of buying the best automobile at the cheapest price, you 
would have an impossible assignment. Your fulfillment of the 
job could vary from locating a car whose owner would give it 
to you free if you would tow it away to finding an excellent 
luxury limousine that sells for tens of thousands of dollars. 
The free car might satisfy the "cheapest price" portion of 
the assignment (unless you could find soneone that would pay 
you to remove a junk car) and the luxury limousine mi^t 
satisfy the "best automobile* part of the project. 

It is essential to fix or hold constant one 
of the variables. The assignment then becomes "Purchase the 
best automobile you can for less than $2,000" (cost is con- 
stant), or "Purchase the best 1932 four-door Model A Ford 
regardless of the cost" (object is constant). 

In education, more and more districts are 
fixing the educational objectives. Limited, specific, 
measurable objectives are established for each educational 
program. The objectives may be 50 per cent of the third grade 
pupils shall increase reading achievement by one year. . . ; or 
80 per cent of the eighth grade pupils shall score 75 per 
cent or higher in mathematics. . . etc. 

By holding the objective constant, different 
methods of achieving the objectives can be compared in terms 
of cost. When one method is less expensive than other methods 
and produces equal or better results, a strong case exists 
for utilizing the least expensive method. Likewise^ when 
the cost is held constant and unequal educational results are 
obtained, the alternative producing the greatest educational 
outcome deserves first consideration. 

Utilizing analysis of this nature to compare 
alternative methods of achieving educational objectives is 
cost effectiveness analysis. in concept, it is deceptively 
simple; in practice, data collection for accurate analysis 
can be quite difficult. 

Measuremen t. Another major component essential 
to an operative cost effectiveness analysis is measurement. 
At least four key steps are important in evaluation: 

1. Determining what is to be accomplished; 

2. Agreement on the degree of accomplishment; 

3. Concurrence on the measuring instruments 
and methods; and 

4. Deciding on how to analyze the results. 

Much has been said and written by professional 
educators about the inadequacy of academxc achievement tests. 
TVhile agreeing whole-heartedly to the short comings, we should 
none the less recognize that achievement testing is still a 
valuable source of dai-a and should be one of the devices 
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utilized to measure school outputs. Other measures such as 
teacher-made tests, opinion scales, check lists and question- 
naires also should be used. A functional management system 
utilizing performance objectives requires that the measuring 
device be identified along with a statement of anticipated 
accomplishments . 

Obviously, there are some problems related 
to measurement. The productivity of a school or a district 
Cannot be quantified completely on the basis of test scores. 
Many non-cognitive segments of the educational program will 
likely remain unverified. Behavior of pupils, attitudes of 
students, school's social atmosphere, and a host of other 
areas may depend upon professional judgments of staff for 
evaluation. 

Further, the tests themselves lack perfection . 
Their content may not parallel the curriculum or they may be 
unfair to the students because of the students* language or 
cultural background. Despite these acknowledged deficiences, 
since measurement is essential, educators must use what data 
is available as a starting point and work to improve the 
measuring devices. 

Nor is cost effectiveness analysis designed 
to replace the decision maker. Value judgments still must 
be made. Cost effectiveness simply provides more basis, 
more data for decisions. 

Program Costs . A further requirement of cost 
effectiveness analysis in education is program costs. 

In the past, school district budgets have 
been developed around budget categories according to functions- 
administrative, instruction, health services, etc. Changes 
from one budget to the next were generally incremental, 
reflecting pupil enrollment increases or decreases, cost of 
living fluctuations, revenue changes, and political appeal. 
The present system of school budgeting (PPBS) requires 
allocation of resources by program — reading, mathematics, 
English, etc. Budgeting and accounting by program is neces- 
sary for cost effectiveness analysis. 

Program budgeting and accounting would need to 
be done uniformly throughout the state if comparisons are to 
be made between school districts. And a uniform school 
accounting system should be maintained throughout the nation 
to draw comparisons between and among the states. 

Thus, it is desirable that a uniform rchool 
budgeting and accounting practice be utilized in the fifty 
states of the U. S. and that this sytem be on the basis of 
program budgeting and accounting. 

Variable Elements . The last component of cost 
effectiveness analysis is a set of variable elements. In 
fact, practically all inputs into the analysis are variable 
elements. Certainly students differ from one group to the 
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next; teachers vary in ability and skills; resources are 
allocated in various wi\ys; and the learning environment changes. 

Many of the changeable elements can be deter- 
mined by decisions. TTie choice may be made to buy new desks 
for the room instead of textbooks. One teacher may have 35 
pupils while another teacher has 25 pupils with a teacher 
aide. One school building may be old and delv^pidated, perhaps 
ex-army barracks, while? an adjoining building is new with 
carpeting, air conditioning, and all modern design. There is 
no reasonable limit to the number of possibilities for varia- 
tions that could exist in the classrooms of a district. 

Cost effectiveness analysis simply takes 
whatever variation that exists and utilizes the program cost 
established for tl^.at situation and relates that cost to the 
effectiveness achieved. Table I depicts this process. 



TABLE I 



COST EFFECTIVENESS ANALYSIS 



Variable Elements: 

Students Teachers 
Environment Equipment 
Materials Supplies 
Resources , 

\ / 



COST 



Educational 
Program or 
Obiective 



/ 1 



EFFECTIVENESS 



Cost Effectiveness Relationship . An operative 
system as has been discussed here would provide information 
for performing effectiveness analysis. The relationship 
between cost and effectiveness may be very simply shown by 
using a formula such as 

C 

Cost Effectiveness = 



I (N) (P2 - Pi) 

where C is Cost of the objective; I is the importance of the 
objective rated on a scale of 0 to 1.00; N is the number of 
pupils; ?! is the per cent of pupils achieving the objective 
before instruction and the per cent of pupils achieving the 
ob j ective after instruction. 
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For example, suppose the following information 
were known about two first grade classes whose reading objec- 
tive was to read at second grade level on a standardized 
achievement test by the end of the year: 



A. Lincoln School using 
Reading Program X 
28 pupils in class 
2 pupils reached objective 
pre-test 
14 pupils reached objective 

post-test 
$1,000 cost 

Reading judged . 95 importance 



B. Washington School using 
Reading Program Y 
24 pupils in class 
1 pupil reached 

ob j ective pre-test 
15 pupils reached 

obj ective post- test 
$1, 500 cost 
Reading judged . 98 
importance 



To compute the cost effectiveness for each 
school for the first grade reading programs, substitute the 
data measured and determined into the formula: 



A. 



B< 



Lincoln School-Program X 
Cost Effectiveness = 



$1,000 



95 (28) (.50-. 07) 

CE = $87.43 



CE = $1^000 
11.438 

Washington School-Program Y 
Cost Effectiveness = 



CE = 

14.818 



98 (24) (.67-. 04) 

CE = $101.23 



The analysis shows that Lincoln School using 
reading program X is better in terms of cost effectiveness. 
Program Y at Washington School costs more per unit of reading 
accompl i shmen t . 

A study of the cost effectiveness analysis 
shows that the variable elements (student population, teachers, 
etc.) were converted to: 1) costs of reading program by the 
budgeting and accounting system; and 2) effectiveness of 
reading program by holding the objective constant and measur- 
ing results. The final step establishes a relationship 
between the cost and the effectiveness of the first grade 
reading. 

Cost effectiveness analysis may be used in a 
variety of ways to help decision makers. In the example, 
different reading programs were compared. The analysis nnay be 
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used to compare the effectiveness of the same programs for 
different groups of pupils — minority pupils with anglo; or 
handicapped pupils with regular pupils. The analysis can 
be used to compare one district with anotiier; one school 
vith another; one grade level with another; or one class- 
room with another. Cost effectiveness can be utilized with 
support programs such as transportation and cafeteria as 
well as with instructional programs. 

Cost effectiveness analysis is one more 
method to provide additional data to th"^ educator for 
effective decision m.^king* 
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STATUS OF COST EFFECTIVENESS AS A METHOD OF ANLAYSIS IN EDUCATION 



Richard B. Home 



(Richard B. Home is Assistant Superintendent- 
Business and Fin.^.icial Services, Office of Los 
Angeles County Superintendent of Schools. He 
received his A. B. degree fron the University of 
California, Los Angeles, axxl his Doctorate fron 
the University of Southern California.) 



For purposes of illustrating the status of the concept of cost 
effectiveness as a method of analysis for aiding decisionnnaking, let us look 
at the two vjords— COST EFFECn\'ENESS~separately. Take the word cost and 
iiTBgine a four point rating scale with the letter "C" being the lowest point 
of achievement and "T" the highest point of achievement. 

C / 0 S T 

Obviously, there is no objective way of determining exactly 
where we are along the continuum of developing widely accepted and used 
methods of determining the real cost of educational programs. However, in 
my opinion, we are sornewhere between the "C" and the "0" on our hypothetical 
scale. Someone might quickly reply, ^'you mean to tell me we are only one- 
quarter way there in our ability to determine the cost of an educational 
program. After all, it is only a matter of adding up the dollars of each 
program by the objects of accounting (salaries, supplies, equipment, etc.)." 
I wish it were that simple--^y answer is yes and that estijiHte my be overly 
optimistic. 

Cost is a measure of the resources that go into an educational 
program. An effective cost system must have at least three essential elements. 

(1) A program oriented budgeting and accounting system using 
educational programs as the primary element for developing a budget and 
determining costs . 

(2) A program budgeting and accounting system uniformly 
applied and administered by qualified personnel at all levels of the 
educational organizational structure (district, intermediate, state, and 
federal. ) 

(3) An electronic data processing system to facilitate the 
timely and accurate collection of data with the capability of nanipulat^ng 
the data to produce meanixigful information. 
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Let us lock at the California scene as a case in point. 
California, like most states, still has a budgeting and accounting system 
based upon the functions of education (administration, instruction, food 
ser^^ices, transportation, etc . ) not programs * A uniform progrom budgeting 
and cost accoui'^ting system has been developed and was scheduled for field 
testing during 1972-73 and operational status during 1973-74. However, it 
ha.s become embroiled in the politics of the legislature and the State Board 
of Education and ^as been delayed at least a year. This represents valuable 
tine lost because any effective cost accounting system applied on a uniform 
statewide basis needs two or three years of operation to work out the pro- 
blems. In the meantime, at the local level sotks school districts are ex- 
perimenting with program budgeting and accounting systems, but the practice 
is not widespread. The competence of the personnel involved in the various 
districts varies and considerable upgrading is necessary, not to ment:''. the 
additional problems associated with far too many srrall administratively in- 
efficient school districts. 

Coming back to my basic point — ^we are only one-quarter of the 
(*ay toward developing and implementing an effective cost system. 

Assuming we had a viable cost system inplemented , what about 
the other word effectiveness? I am sorry to report the degree of progress is 
even less. If we take the word effectiveness and treat it in the same manner 
as cost and imagine a thirteen point mting scale with the first letter 'T'' 
being the lowest point of achievement, in my opinion we are somewhere between 
the two "F's". In other words, only two thirteenths of the way along a 
thirteen point scale. 

ef/fectivenes,s 

Adding this "effectiveness" dimension to our method of 
analysis really complicates the concept fa?: beyond the complexity of a cost 
system. Whereas cost is a measure of the resources that go into an education- 
al program, effectiveness measures what ccmes out or the result — to what ex- 
tent were the program objectives achieved. Even if we develop a successful 
method of cost analysis, it will provide barren data rather than meaningful 
infonriation for decision-making. We shall be able to say with a high degree 
of confidence tl^t a particular educational program l^s a cost of blank 
dollars, but we shall not be very sure how effective the program is in terms 
of educational perf orrrance . 

Educational research has an important role on the effective- 
ness side of our two word method of analysis. An all out effort should be 
made toward developing instruments and methods for measuring educational per- 
fonrance. Since at the present time we are only spending about three tenths 
of one per cent on educational research and development, I would like to 
support an immediate substantial increase in the amount of money spent for 
that purpose even at the expense of curtailing other programs. In my opinion, 
a high level strategic decision should be made in that direction and the loi^ 
term results will justify the higher priority and cost. 
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terns of the specific goals of individual principals, ar, well as taking 
into account the limitations to which the principals are r abject. 

8, Greene, Robert E. Administrative Appraisal t A Step to Improved Leader - 
ship , National Association of Secondary School Principals, Washington, 
D.C., 1972. 48 pages. 

Weaknesses in many of the current appraisal systems are pointed out, 
followed by general guidelines for designing or redesigning a goal- 
oriented system for a district's particular needs. 

9. Jones, Anthony S. *'A realistic approach to teacher evaluation," Clear- 
ing House , 46:474-461, April 1972. 

Includes teacher evaluation forms. 

10. Klein, Stephen P. and Marvin C. Alkin. "Evaluating teachers for outcome 
accountability," UCLA Evaluation Comment , 3:5-11, May 197^. 

Dfrscribes possible ways to implement teacher-evaluation laws. 

11. Lambert, M. Dale. "Hefocusing. teacher evaluation: a process of guided 
self-analysis,'* Thrust for Education Leadership , (ACSA) 1:41-44, Feb. 
1972. 

12. McNeil, John D. "Concomitants of using behavioral objectives ir. the 
assessment of teacher effectiveness," pages 47-55 of Current Research 
on Instruction , edited by Richard C. Anderson and others. Prentice- 
Hall, Englewood Cliffs, N. J., I969. 396 pages* 

15. Marshall, Max S. "Reverse grading," F/iucational Leadership , 28:665-665, 
March 1971. 

The writer feels that student appraisal of teachers will not bring 
progress or peace. 

14* NEA. Research Div. "The evaluatee evaluates the evaluator," ERS 
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Circular 1970. 52 pa^es. 

A survey of "client-oriented" evaluation. 



15« ^. "Eva.'^.uating administrative/supervisory performance/' £RS Circu - 
lar //6, 1971 . 60 pa^es. 

16. . "Evaluating teaching performance/' ER3 Circular #2, 1972. 60 

pa^es. 

Gummarizes the results of an Educational Research Service survey. 

17. Ober, Richard L. , Ernest L. Bentley, and Edith Killer. Systeinatic Ob- 
Ger'/ation of Teaching » An Interaction Analysis *" Instructional Strategy 
Approach . Prentice-Hall , Englewood Cliffs, N. J., 1971. 256 pages. 

Accountability, both for the actions of the teacher and the continuous 
interactions within the classroom, by means of systematic observation. 

18. Owens, Mary Seymour. "Evaluation of teachijng competence by three groups 
of educators," Journal of Experimenta l Education , 40:77-82, Winter 1971* 

A study "to compare the perceptions of teacher competence within and 
air^ong three groups of educators (teachers, college supervisors, and 
public school administrators) as measured by the interview scales of 
IOTA." 

19. Pi Lambda Theta. The Evaluation of Teaching . A Report of the Second 
Pi Lambda Theta Catena. Washington, D.C., I967. 259 pages. 

Different criteria for evaluating teaching, including pupil outcomes. 

20. Plowman, Paul D. Behavioral Objectives : T eacher Success Throu>?h Stu- 
dent Performance ^ Science Research Associates, Chicago, 1971. 1B8 
pages. 

2-. Popham, W. James. Designing Teacher Evaluation Systems . Instructional 
Objectives Exchange, Los Angeles, Cal., Dec. 1971. 49 pages. 

*'A series of suggestions for establishing teacher asses*5ment proce- 
dures as required by the Stull Bill (AB 295), 1971 Califc Tiia Legisla- 
ture." (Subtitle) 

22. Redfem, George B. "Client-centered evaluation," School Administrator , 
pages 7-10, March 1972. 

25» • "Evaluating administrative productivity: can it be done/" 

School Administrator , pages 15-16, July I971. 

The author say£? "yes" provided certain prerequisites exist. 

24. Hoberson, E. Wayne. "Teacher self-appraisal: a way to improve instruc- 
tion," Journal of Teacher Education , 22:469-47$, Winter 1971. 

25 • Rosenshine, Barak. "Enthusiastic teaching: a research review," School 
Review , 78:499-514. Aug. }970. 

A review of the attempts to assess the relationship of enthusiasm to 
pupil achievement and to specify the behavioral components of enthusi- 
asm, 
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26. Rosenshine, Barak, and Norma Purst. "Research on teacher performance 
criteria,** pages 57-72 of Research in Teacher Educatio n; A Symposium , 
edited by B, Othanel Smith for the American Educational Research Associa- 
tion. Prentice-Hall, Englewood Cliffs, N. J., 1971. 166 pages. 

27. Rosenshine, Barak. Teaching behaviors related to pupil achievement: a 
review of research," pages ^^^^8 of Research Into Classroom Processes ; 
Recent Developments and Next Steps , edited by Ian Westb^iry and Amo A. 
Bellack, Teachers College Press, N. Y., 1971. 

28. . Teaching Behaviours and Student Achievement . National Fotmda- 

tion for Educational Research in England and Wales, London, 1971. 229 
pages. (Available from FexTihill House Ltd., 303 Park Ave. South, New 
York, N.Y. 10010). 

A summary of more than 50 studies in which some measure of observed 
teacher behavior was related to one or more measures of student achieve- 
ment. 

29. Schalock, H. Del. "The focus: knowledge, teaching behavior, or the 
products?" pages 45-49 of Performance-Based Certification of School 
Personnel, edited by Joel L. Burdin and Margaret T. Reagan. ERIC Clear- 
inghouse on Teacher Education, Washington, D.C., 1971 • 140 pages. 

50, Simpson, Ray H. Teacher Self-Evaluation . Macmillan, N. Y., 1966. 100 
pages. 

51. U.Sc Office of Education. Do Teachers Make a Difference? A Report on 
Recent Research on Pupil Achievement . USOE, Washington, D.C., 1970. 
181 pages. 

Contributors: Alexander M. Mood, James W. Guthrie, Henry M. Levin, 
Eric Hanushek, George W. Mayeske, Stephen Michelson, Robert M. Gagne, 
and James S. Coleman. (See also item #32. ) 

32. . How Teachers Make a Difference . USOE, Washington, D.C., 1971. 

166 pages. 

An assessment of the present state of the art of teaching. Papers by 
Alexander M. Mood, Philip W. Jackson, N.L. Gage, Dan C. Lortie, Barak 
Rosenshine, Ned A. Flanders, and Lawrence M. Stolurow. (See also item 
#31.) 

53 • Yeldman, Donald J. "Pupil evaluation of student teachers and their su- 
pervisors," Journal of Teacher Mucation» 2!: 165-1 67 <i Summer 1970. 

34 • Washington, Eva. Expert Teacher Acti ^.>r. Program . Fearon Publishers, 
Belmont, Cal., 1971* 115 paf?es. 

Based on a set of standai^ds, 25 variables, that leads to a definition 
of what constitutes expert teachingu 



111 



ERIC 



Ill MEASUREMENT OF STUDEIJT ACHIEVEMENT 



1, Beall, Lewis L. "Pupil progress: one measure of teaching," California 
School Boards , ^1:10-15, May 1972. 

Includes the hazards as well as the advantages. 

2. Block, James H. "Criterion-referenced measurements: Potential," School 
Review, 79:289-298, Feb. 197I. 

The author examines the "alleged flaws" pointed out by Ebel (see item 
//6) and defends the use of such measurements because of their potential 
"to promote the learning of all." 

5. Bloom, Benjamin S., J. Thomas Hastings, and George F. Madaus. Handbook 
on Formative and Summative Evaluation of Learning . McGraw-Hill, N. Y., 
1971. 923 pages. 

The intent of this book is to help the teacher bring students up to 
mastery levels of learning, use evaluation for instruction decisions, 
understand and use evaluation techniques for cognitive and affective 
objectives, and cooperate with specialists and others in evaluation. 
The second part of the book deals with evaluation in each of the major 
subject fields and levels of education. 

4. Bloom, Benjamin S. "Testing cognitive ability and achievement," pa^^es 
579-"$97 of Handbook _of Research on Teaching , a project of the American 
Educational Research Association, edited by N. L. Gage. Rand McNally 
(Sc Co., Chicago, 1963- 1218 pages. 

5. Darlington, ^^ichard B. "Another look at 'cultural fairness'," Journal 
of Educational Measurement , 8:71-82, Summer 1971 • 

Four definitions of "cultural fairness" are examined. 

6. Ebel, Robert L. "Criterion-referenced measurements: Limitations," 
Gchool Review, 79:282-288, Feb. 1971. 

The author states that "the us'j of criterion- referenced measurements 
cannot be exoected to improve significantly our evaluations of education- 
al achievement." (See also item //2.) 

7. Green, Lonald Ross. Racial and Ethnic Bias in Test Construction . CTB 
McGraw-Hil] , Del Monte Research Park, Monterey, Cal. Adapted from Racial 
and Ethnic Bias in Test Construction , Final report of the U.S. Office of 
Education, Contract No. OEC-9-70^0058 (057). 

8. Husek, T. R. "Different kinds of evaluation and their implications for 
test development," UCLA Evaluation Comment , 2:8-10, Oct. I969, 

9. John, Vera P. and Vivian M. Homer. "Testing and evaluation procedures," 
pages 142-I65 of Early Childhood Bilingual Education , by Vera John 
and Vivian M. Homer, Modem Language Association of America, N. Y., 
1971. 187 pages. 
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10. Johnson, Geraldine F. "Metropolitan tests: inappropriate for ESEA pu- 
pils," Integrated Education t 9:22-26, Nov/Dec. 1971. 

The author rejects the assertion that Metropolitan Achievement Tests 
provide valid leading ability measurement of ESEA Title I students. 

11. Karmel, Louis J. Measurement and Evaluation in the Schools . Macmillan/ 
Macmillan-Collier, London, 1970. 492 pages. 

In addition to the technical aspects of testing and measurement ap- 
plications, the contemporary issues and problems of testing are dis- 
cussed, such as IQ, heredity and environment, minority testing. 

12. Khatena, Joe. "Some problems in the measurement of creati/e behavior," 
Journal of Research and Development in Education , 4 j 74^82, Spring 1971 • 

^3• Klein, Stephen P. "Evaluating tests in terms of the information they 
provide," UCLA Evaluation Comment , 2:1-6, June 1970. 

14» • *'The uses and limitations of standardized tests in meeting the 

demands for accountability," UCLA Evaluation Comment , 2:1-7f Jan. 1971. 

15. Mathis, Harold I. "The disadvantaged and the aptitude barrier," Per - 
sonnel and Guidanc e Journal , 47^467-472, Jaxij 1969* 

Aptitude testing is contrasted with achievement testing, particularly 
in reference to underlying assumptions, validation procedures, and in- 
terpretation of test scorese 

16. Mayo, Samuel T. "Mastery learning and mastery testing," NOME Measure - 
ment in Education , 1:1-4, March 1970. (National Council on Measurement 
in Education, Special Report series). 

Mastery learning can be characterized as a "method of organizing dis- 
crete educational objectives which are meaningful and useful to the in- 
dividual." The importance of mastery testing of mastery learning is 
discussed in this article. 

17 • Moreno, Steve. "Problems related to present testing instruments," 
El Grito, 5:25-29, Spring I97O. 

Problems of validity and reliability of English test scores on Span- 
ish-speaking or bilingual children^ and of availability of valid tests 
for Mexican- American children. (El^ Grito : _A Journal of Conj;emT)orary 
Mexican-American Thought . P. 0. Box 9275, Berkeley, Cal. 94709; 

18. Moxley, Roy A. "A source of disorder in the schools and a way to re- 
duce it: two kinds of tests," Educational Technology , Teacher & Tech - 
nology Supplement , 10:S3-S6, March 1970. 

Norm- referenced and criterion- referenced tests are viewed as to vari- 
ability, construction, methods of instruction, objectives and The pt/od- 
uct. 

19» Popham, W. James, ed. Criterion-Referenced Measurement : An Introduc - 
tion . Eaucational Technology Publications, Englewood Cliffs, N» J., 
1971. 108 pages. 
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Includes "Instructional technology and measurement of learning out- 
comes: Some questions," by Kobert Glaser (Reprinted from Am erican 
Psychologis tt 19^5) , as well as papers by Popham and Husek:"Tsee iterc 
//20)f Garvin, and Cox. 

20. Popham, V. Janies, and H. Husek. ^^Implications of criterion-refer- 
enced measurement,'* Journal of Educational Measurement , 6:1-9, Spring 
1969. 

Norm-ref erenced, and particularly criterion- referenced, measurements 
are examined with respect to variability, item construction, reliabil- 
ity, va.lidity, item analysis, reporting, and interpretation. (See 
also item #19*) 

21. Shoemaker, David M. "Criterion-referenced measurement revisited," 
Educational Technology , 11:61-62, March 1971. 

The author says it might be more appropriate to refer to "criterion- 
referenced" tests as "achievement" tests. Their use should be judged 
according to the ajnount of useful information they provide the class- 
room teacher. 

22. Spence, Allyn G,, and others. "Home language and performance on stan- 
dardized tests," Elementary School Journal , 71:509-515, March 1971. 

23. Stone, Chuck. "Testing and the educational power struggle," Integrated 
Education , 9:4-9f July/Aug. I97I. 

The author, who is a staff member of the Educational Testing Service, 
discusses the testing experience as it affects the Mack community. 

24. Thomdike, Robert L. "Concepts of culture-fairness," Journal of Educa - 
tional Measurement , 8:63-70, Summer 1971. 

25. Thresher, B. Alden. "Uses and abuses of scholastic aptitude and 
achievement tests," pages 24-40 of Barriers to Higher Education , College 
Entrance Examination Board, N. Y., ^^^^^ I5I pages. 

26. Williams, Frank E. "Assessing pupil-teacher behaviors related to a 
cognitive-affective teaching model," Journal of Research and Develop - 
ment in Education , 4:14-22, Spring I97I. 



IV CTALUATIOiNi OF CURPiICIjLUM 
AND EDUCATIONAL PROGRAMS 



1. Bloom, Benjamin S., ed. Taxonomy of Educational Oh.iectives . Handbook 
_I: Coigrnitive Domain . Longmans, N. Y. , 19^6. 207 pages. 
Includes illustrative tests. 
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2. "Evaluating educational programs: a symposium," Urban Review , 52 14-22, 
Feb. 1969. 

Evaluation of Title I. Contributors: J. Wayne Wrightstone, James S. 
Coleman, David G. Hawkridge and Albert B, Chalupsky, Henry S. Dyer, 
John Mann, Martin Mayer, Edward A, Suchman, Peter H, Rossi, Edward 
Wynne, and Michael Scriven. 

3« Forehand, Getrlie A, "Curriculum evaluation as decision-making process" 
Journal of Research and Development in Education, 32 27-57, Summer 1970. 

4. Cooler, Dennis D. and Arden D, Grotelueschen. "Curriculum development 
accountability," Educational Leadership , 29:165-169? Nov. 1971. 

The role of evaluation in curriculum development. 

5. Harbeck, Mary B. "Instructional objectives in the affective domain," 
^Educational Technology , 10:49-52, Jan. 1970. 

The author says usually educational goals in the affective domain 
have been stated in the school philosophy and have been forgotten in 
the planning of the actual instructional planning. How to state and 
measure the outcomes is a major problem in this domain. 

6. Krathwohl, David R., Benjamin S. Bloou, and Bertram B. Masia. Taxonomy 
of Educational Objectives . Handbook II : Affective Domain . David 
McKay Co., N. Y., I964. 196 pages. 

The classification of educational goals. 

7. Mager, Robert F, Preparing Instructional Objectives . Fearon Publish- 
ers, Palo Alto, Cal., 1962. 60 pages. 

8. Ontario Institute for Studies in Education. Curriculum Theory Network 
Monograph Supplement, "Curriculum evaluation: potentiality and reality," 
8/9f 1971-72. Iikiited by Joel Weiss. Ontario Institute for Studies in 
Education, 252 Bloor St., West, Toronto 181, Ontario^ Canada. 256 pages. 

Evaluation of materials, instruction, and outcomes of curriculum 
programs. 

9. Wight, Albert R. "Beyond behavioral objectives," Educational Technology 
12:9-14, July 1972. 

Evaluation of programs based on behavioral objectives. 
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APPENDIX 



This paper was delivered at last year's conference, the 23rd 
Annual State Conference on Educational Research, on November 19 > 
19 71* It was not published in the Proceedings {EeseoToh Besume 
Number 46) , but we are including it at this time because of its 
appropriateness to this year's theme of Evaluation. 




ACCOimTABlLlTY IN INSTFJJCTIONAL 



PRODUCT DEVELOPMENT 



Harry Handler 
Chairman, Department of Instructional Technology 
School of Education 
University of Southern California 

A discussion of accountability In instructional product 
development will encounter as much if not more £.mbiguity as have 
other efforts to apply the concept of accountability to existing 
activities in the educational community. As Dick Harsh has 
already indicated: "The fascinating thing about labels is that 
they are widely used by persons having diverse understandings 
and attitudes toward the entity which the label represents.'* 

Each member of the audience will interpret the comirents 
which follow in terms of his past experiences and anticipated 
needs. For example, last week four students were discussing the 
results of the State Testing Program. One of the students was 
a sociology major, another an ecology major, the third a journal- 
ism major, and the fourth had not selected a major. 

The sociologist: Well, just as we have been predicting, 
IQ's are decreasing in urban areas. 

The ecologist: You see, students who live in smog- 
filled environments are losing IQ points. 

The journalist: Wowl Relationship found between 
decreasing IQ's and increasing smog. 

The non-major: Why be surprised? Anyone who continues 
to live in a setting where he has to breath the 
stuff can't be too bright. 

In the Rand Publication, Accountability, Program Budgeting , 
and the California Educational Information System: A Discussion 
and Proposal , Farquhar identifies the following three areas for 
which educational personnel are held accountable: 

1. Education planners are judged on (a) whether or not the 
chosen goals are desired by the school board, commun- 
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ity, and state, and (b) whether or not the programs 
established effectively meet these objectives. 

2. Educational Administrators are judged on their 
ability to monitor and administer the programs so 

as to bring about successful outcomes (if objectives 
and program design are sound). 

3. Teaching personnel and related administrators are 
judged on their ability to bring about the behavioral 
and educational outcomes stated in the objectives 
and implicit in the educational program structure. 

Accountability in instructional product development is 
concerned with all three of the above areas. As the staff of the 
Southwest Region^il Laboratory has demonstrated, instructional 
product development includes: 

1. The production of research-based systems to accomplish 
specified instructional outcomes under natural 
conditions . 

2. The operational use of a program, without further 
direct assistance from the developer, in all the 
appropriate classes in a school district . 

3. The continued use of a product to consistently obtain 
Che instructional outcomes at a specified level of 
effectiveness . 

This presentation will partially respond to the concerns 
expressed by Harmes when he raised the following question: 
"Why should educators, particularly public school educators, be 
singled out and be asked to guarantee results from instructional 
programs which are unreliable to varying degrees?" 

Four basic subsystems and related evaluation criteria in 
terms of subsystem objectives required to successfully accomplish 
specified instructional outcomes are presented, in addition, 
three distinguishable phases of the development process are 
reviewed . 

The content of this paper draws heavily on the work of 
the staff of the Southwest Regional Laboratory and will be 
available in greater detail in the forthcoming publication 
Instructional Product Development . Baker, Robert L. and Richard 
E. Schutz, editors. Southwest Regional Laboratory for 
Educational Research and Development. 
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Four Subsystems and Related Evaluation Criteria 



in Terms of Subsystem Objectives 



Instructional systems . An instructional system refers 
to the research-based methods and materials prepared to accomplish 
specified instructional outcomes under natural conditions. The 
instructional system includes everything needed by the schools 
to obtain specified outcomes. A system, therefore, includes 
specification of instructional outcomes, student materials, 
inscructor procedures, delivery mechanism for delivering the 
student material to the student, performance indicators which ara 
used for evaluating student progress, for diagnosing student 
learning difficulties, and for Instructor and instructional system 
accountability . 

Instructional systems evaluation criteria in terms of 
system objectives : 

1. Statement of anticipated observable student outcomes. 

2. Criterion measures to determine the accomplishment 
of the outcomes . 

3. Student instructional materials. 

4. Statement of learner prerequisites in terms of initial 
proficiency the learner must exhibit. 

5. Statement of the teacher's instructional respon- 
sibilities . 

6. Evidence that the system has yielded dependable 
results . 

7. Data concerning instructional time and study time 
requirements . 

8. Statement of direct and indirect costs. 

It is unreasonable for the developer to expect that a 
given instructional system will achieve all of the intended 
objectives during the first field tryont. As Deterline has 
stated, *Ve must rely on data. Too many courses are considered 
to be 'finished, polished, and perfect,' because a group of 
subject matter experts or instructional specialists pronounced 
them so. In their best judgment, the course SHOULD teach. An 
empirical approach requires more than that. The subjective, 
intuitive ano exper ientivil judgments of the experts are interesting^ 
encouraging and nice to have. But the final judges are the 
students. The data they produce as they proceed through the 
various components of the course, and the evaluation data, and 
any available follow-up data tell the story. We are accountable 
for the data; if the data are not adequate in comparison with the 
data we had hoped for, we have not met the requirements." 
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The developer must collect the data essential to provide 
information related to the system objectives along with data 
which provide direction for appropriate revisions. Although 
these data are frequently collected, inaoequate planning many 
times does not allow for the time, resources, and related costs 
required to conduct the subsequent iterations. 

Unfortunately, the impression exists that when the 
developer has established some arbitrary criterion level of 
acceptable performance that he is willing to abandon the 
critical task of effecting learning for those who do not meet 
the criterion. This is not the position taken by the respon- 
sible developer. However, he has learnea that there is a 
direct relationship between the levels of difficulty and mastery 
desired and the development costs. 

In the November-December, 1967 issue of the Harvard 
Business Review, Abraham Zaleznik dirscussed the relationship 
between the individual's ability to cope with disappointment 
and related success in government and industry. The instruc- 
tional product developer must also learn to manage disappoint- 
ment in a constructive manner and to avoid the pitfalls 
inherent in the use of rationalization, projection, or 
compensation. If the objectives of an instructional system are 
not attained, he is not being accountable if he resorts to 
such statements as: 1) "It doesn't make any difference, the 
people who are demanding the data won't understand it anyhow"; 
2) "The materials are fine, the schools goofed it up"; or 3) 
"The results may not be too good, but we have learned how to 
produce the materials cheaper than anyone else". 

If the data do not support the objectives, the developer 
must responsibly analyze the sources of weakness and assume 
responsibility for the improvement of the product. 

Training systems . A training system refers to the 
materials and procedures required to train persons who are in 
direct contact with the pupil and have the ultimate resoonsibllity 
for various phases of instruction. In addition to the teacher, 
these human resources include parents, tutors, aides, etc. The 
training system must be designed in such a way as to effectively 
train all personnel involved in the teaching process in a 
specified manner consistent with the instructional requirements 
of the instructional system. 

Training systems evaluation criteria in terms of system 
objectives : 

1. Statement of personnel requirements and inter- 
relationships (e.g., teachers, tutors, parents, 
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aides , etc . ) . 

2. For each personnel category identified, proficiency 
anticipated at the end of training and in conducting 
the instructional program, 

3. Measurement procedures to determine the accomplish- 
ment of the training outcomes . 

4. Instructional materials for each category of 
trainees . 

5. Instructional materials for trainers, 

6. Feasible alternative plans for sequencing and 
scheduling training for each trainee category. 

7. Evidence that the system has yielded dependable 
results . 

8. Statement of direct and indirect costs. 

The measurement procedures to determine the accomplish- 
ment of the training outcomes are not the same as the measurement 
procedures to determine pupil performance. Pupil performance 
measures are not a direct means of determining training system 
outcomi^.' . Training system evaluation criteria should be 
directly related to the unique objectives listed. 

The economic and political contraiuts of the school 
community should be considered in the development of the 
training requirements. Training systems which require instant 
and dramatic modifications of existing scheduling practices, 
teaching procedures, and personnel policies will be less 
effective than those which recognize existing competencies and 
build on associated strengths. 

Deterline has strongly stated the point as follows: 
"So we can conclude that both teachers in educational settings 
and instructors in training settings might be expected to cry 
'foul and unfair' when faced with the notion of accountability. 
And I don't blame them a bit. Unless someone else accepts 
accountability for teaching those teachers relevant skills 
beyond thcr^^ s&hcy already possess, and unless the conditions 
that limit their effectiveness can be changed, then there is no 
justification for expecting them to do better, or for holding 
them accountable for doing so. They need more effective 
techniques, procedures, methods, materials and other components 
of instruction. And then, when new components are added, no 
matter how innovative, how effective, there will be a limit to 
the overall effectiveness of that totality, probably still short 
of that ideal of all students learning everything. That new 
limit will have to be determined empirically, and then additional 
research and development efforts can be aimed at raising that 
ceiling. 
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Installation systems . An installation system consists 
of the procedures and materials required by a local educational 
agency to effectively introduce an instructional program. It 
provides for those considerations related to the administrative 
authority and responsibility for supervisory service, curriculum, 
and pupil personnel services. This system includes briefing 
information about the instructional program, procedures for 
providing various categories of agency personnel with pupil 
performance data in a form they find manageable and useful, and 
materials which can be used for public information purposes. 

Installation systems evaluation criteria in terms of 
system objectives : 

1. Materials for uce by installing agency to describe 
a program at a "public information** level suitable 
for at least the following audiences: governance 
group 3 operating staff, general public. 

2. Statement of feasible procedures for procuring and 
storing instructional and training system materials. 

3. Statement of anticipated required revisions in 
prevailing personnel and/or administrative policies. 

4. Statement of feasible procedures for required 
personnel assignment and scheduling. 

5. Statement of feasible alternatives for training 
of training-supervisors . 

6. Statement of anticipated administrative requirements 
for maintaining the program. 

7. Statement of alternative options where the new 
program interfaces with remaining extant programs. 

8. Evidence thp.l the system has yielded dependable 
results . 

9. Statement of direct and indirect costs. 

The requirements for the mainte le of program out- 
comes are of particular importance. Alt....agh the installation 
of a program may be initially successful, without knowledge of 
the requirements to sustain program performance, the user may 
find that in a very brief period the program will become 
fragmented, diluted, and void of its initial integrity and 
effectiveness . 

Accountability systems . An accountability system 
includes all of the mechanisms required to maintain maximum 
pupil performance and the procedural adequacy of each of the 
three above-mentioned systems: the instructional system, the 




training system, and the installation system. Differentiating 
assignments and establishing responsibilities for which 
development personnel will be held accountable aro anchored to 
specified program and/or procedural outcomes. 

Accountability systems evaluation criteria in terms of 
system objectives ; 

1. Assessment devices for all program outcomes for 
which accountability is to be maintained, related 
both to pupil proficiency and to systems procedural 
adequacy. 

2. A human resources analysis which may be used as 

a basis for differentiating and assigning account- 
able responsibility. 

3. A data collection and reporting procedure for 
maintaining accurate and timely bench r:ark 
information. 

4. Statement of consequences or alternatlv-^s to be 
effected if performance at a given bench mark 
test is determined to be unaccep?:ably low. 

5. Available forms and/or equipment to operate and 
report the bench mark tests. 

6. Evidence that the system has yielded dependable 
results . 

7. Statement of direct and indirect costs. 

The functions of; the accountability system are related 
to process evaluation as discussed by Stuff lebeam et al. Since 
the instructional product developer relies heavily on the 
application of formative evaluation strategies, he must also 
provide a means for the formative documentation of his work. 
A systematically prepared documentation program includes 
procedures related to the planning, reporting, and reviewing 
requirements at each stage of the development effort. 

Similarly, Harmes has written: "Defining the 
objectives of a process performance contract takes on dimensions 
that are different from those of product contracts. As the 
best type of objective for product contracting defines how it 
will be determined that students have learned, the best type of 
objective for process contracting defines how it will be 
determined that a process has been, or is being, operated 
as intended . " 

"Special attention must be given, however, to the methods 
by which performance will be monitored. Products can be and are 
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measured at a point in time. Processes, on the other hand, 
take place over a span of time. Unless one wishes to contin- 
uously monitor an instructional process, a very time-consuming 
and expensive thing to do, he must devise techniques whereby 
the process is sampled at points in time or during saoiTt time 
spans. From this sampling, it is then inferred that the process 
taking place between samples is similar to the samples of it." 

The Development Process 

Progress in the development of a product can be described 
in terras of three distinguishable phases: Pre-License, Non- 
exclusive License, and Distribution License. 

Pre-License . Emphasis during the Pre-License phase is 
on the development of instructional and training systems. The 
developer is heavily involved in prototype and component testing. 
Tryouts are of short duration, one day to several weeks. The 
developer provides intensive human and financial support to 
school personnel during these early stages of product develop- 
ment. It is during this period that uncertainty related to 
product specifications, instruction parameters, and instrucCional 
effectiveness is reduced. The materials are not developed at 
the expense of the schools. 

Nonexclusive License . The Nonexclusive License phase 
provides the opportunity for larger scale tryouts of longer 
duration. Emphasis is on the development of installation and 
accountability systems. Quality assurance procedures are 
developed for use by the schools in monitoring product per- 
formance. This is a period of shared cost responsibility 
between the developer and the schools. 

The Nonexclusive License phase also provides the 
developer with an index of user interest and product utility. 
Efforts during this period involve interface activities between 
completed systems and educational programs already operational. 
The assembly of systems into programs and installing the program 
on a wide-scale basis provide new areas for investigation that 
heretofore have received no research and development attention. 

Distribution License , The Distribution License phase 
Involves assisting the private sector, within the constraints 
of the developers mission and resources, to install the 
Instructional and support systems on a national basis. This 
Is the category of the private marketplace and public school 
finance. Although this i£ not a research and development 
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category, specific attention must be given during the earlier 
research and development phases to advance planning of pro- 
curement alternatives if the results of a progranrmatic develop- 
ment effort are ever to be implemented. An important axiom 
is 5 "Don't build it out of reach of the consumer." 

To the individual teacher, without resources, working 
with hand-me-down materials, or materials he has purchased on 
his own, it can be said: "Go to it, more power to you, if you 
can identify what society wants students to learn, how you will 
teach it, and how you will know if your students have mastered 
the content, you are accountable." 

To the task force, given fifty thousand dollars to 
accomplish what others have failed to accomplish given five 
times as much, working day and night to improve the quality 
of instruction, assisting teachers in meeting the objectives 
of the instructional program, forced to use crude measures to 
determine program effectiveness and constantly operating in an 
atmosphere of expediency, it can be said: "Tell it as it is, 
you have nothing to hide, don't oversell, and at the same time 
don't underestimate the contribution you are making." 

And, to the institutions, both public and private, who 
are in position to bring together the resources necessary to 
develop research-verified, reliable, cost-effective, and cost- 
efficient instructional products, it can be said: "Unfortunately, 
not even you can do it alone." 

Stephen Barro has written, "Each participant in the 
educational process should be held responsible only for those 
educational outcomes that he can affect by his actions or 
decisions and only to the extent that he can affect them." 

If we accept Barro's position, then if follows that 
the instructional product developer is directly responsible 
for ensuring that his product leads to the pre-specif ied 
outcomes for which it was designed. But, the further removed 
the materials and procedures are from his control, the less 
he can be held accountable for the effectiveness of the product. 
Therefore, he must accept responsibility for the development of 
the product and be committed to affecting a continuous chain 
of responsibility/accountability for each of the agencies, systems, 
and individuals who use the product- Without this sequential, 
programmatic, sharing of responsibility, it is doubtful that 
the educational community will be able to respond to the existing 
demands for accountability. 
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